Go to the new cbloom rants @ blogspot

12-18-09 | The ACM

There's a bunch of hubbub lately in the game/graphics community about the anti-scientific actions of the ACM.

I find it all much ado about nothing. Of *course* the ACM is a bunch of cocks that are against scientific progress and dissemination of information. Ooo the group of evil bastards who claim all the rights to the paper that you yourself wrote are being dicks !? Well big fucking surprise. Woop-tee-do.

Everyone should boycott the ACM and the IEEE. Duh. The paper publications have absolutely zero purpose now. The conferences have almost no purpose, it's just a bunch of hob-nobbing and back-slapping. The internet is how real information is conveyed. I'm not quite sure why people are so enamored of conferences. I can get way more information in 10 minutes reading papers on the internet than you can get from a 4 day conference, because I am pulling the information I want at the pace that I want, not having it pushed to me.

Now, peer review and collection of papers does in fact provide a service. You don't just want researchers putting up papers on their own web sites with no organization and no peer review. (for example, Arxiv is great and all, but the lack of organization and peer review makes it weak as a primary publishing location).

If you actually cared about getting away from the evil iron grip of the ACM and IEEE bastards, you should work harder to organize and promote the free online journals, of which there are many, but they haven't caught on because the majority of major researchers are smitten by the prestige of the ACM/IEEE names.

Go to DOAJ , pick a journal, support that. Or you know, start a new one focused on higher performance / realtime computer graphics.

12-17-09 | Economics

In the spirit of not just railing against "the system" but trying to maximize within it, let's talk about money.

It seems abundantly clear now that nothing substantial is going to be done about health care at all. This is very bad for most Americans (good for the few who were being denied coverage but will now get it). It's mainly good for the health care industry. It appears health care will continue it's super-GDP growth rate. It seems obvious for me that there are two consequences of that we should care about : 1. Health care stocks will outperform the rest of the market, and 2. the US economy in general (& mainly any industry involving labor) will underperform the world.

(BTW I've seen a lot of blame making within the Democrats about who's to blame for the health care failure - Lieberman is certainly a cock, but the blame is absolutely clear : the Republican block who seem to be intent on fucking up this country)

Similarly, it is now clear that absolutely nothing substantial will be done to reform the financial sector. I find it particularly hillarious the way Obama stands up and tries to brow-beat bankers into "doing more for America". LOL. First of all, we saved those banks, we should just make them do whatever the fuck we want them to do if you had any guts. Second of all, pumping money into the economy is not how you fix it, that's how you make a new bubble. Third, of course the public chastisement is not going to create any real change, all it's going to do is create some empty gestures that look good; laws and regulations are how you really change things.

Anyhoo, again we should think about how we can play this. We should be somewhat optimistic on financial stocks. We should be highly concerned that we are moving into another bubble and crash cycle right now. We should be scared of the value of the dollar.

Randomly I was thinking to myself - who do I actually trust/respect on economics? And the one that came to mind was Paul Volcker ; he's the only Fed Chairman I know of to actually be fiscally realistic and responsible. Anyway, some random searching of him confirms that he is in fact saying good things (and by that I mean saying the same thing as me) :

volcker 1
volcker 2
Sitglitz 1
Sitglitz 2
Sitglitz 3

12-17-09 | Spam Stupidity

A friend send me this email a few days ago, and Gmail wisely decided it was Spam :

"You work on a 64 bit PC at Rad, right? Did you have any weird issues with any dev software? Was it a relatively painless switch?"

This is one of the most obvious examples yet that I've gotten which is just completely retarded.

I've written before about how they should clearly have an exclusion for people who I have extensively emailed with in the past. WTF. (one of the most retarded examples that I've mentioned previously was randomly picking a few mails out of an ongoing thread and calling them spam, but not the others in the same thread)

Clearly another one should be : if there are no links in the email, then greatly decrease the spam decision threshold (eg. call fewer things spam). Only spam with links are dangerous, and it's very rare to get spam without links these days anyway.

Also if there's no mention of banks, money, credit, penises, or viagra, it's probably not spam.

Obviously their spam filter is just broken. But even aside from being broken, I'm sure it's missing the concept of cost/benefit. That is, for a given mail, you need to decide how bad it would be to misclassify it as spam when it's not spam, and vice-versa. The thing is, that cost is not a constant. It should be dependent on the content and the sender. There are some simple cases, like if the content is "harmless" - no links, no attachments, no mention of Nigeria - then the cost of letting through spam is not very high. Once you guess the cost of each outcome, then you can have your Bayesian spam system give you a guess of what % chance this is spam, and you evaluate the EV of each classification and make the maximum EV decision.

11-29-09 | 7 Habits of Highly Retarded People

1. Being annoyed when you say something they "already know". Smart/effective people don't care when you say something they already know. They understand that you're just trying to establish the agreed basis to make the next step. Logical/effective people discuss and learn by establishing firm agreed upon tenets and then going through a series of logical conclusions. Retarded people get all annoyed when you say something obvious cuz they think you're calling them dumb.

2. Saying they "get it" when they don't. This is a huge annoyance that I struggled with when I was a teacher. Smart people will listen and then tell you "hmm I don't get that, how does this follow from this?". Retarded people are really eager to show how smart they are so they want to say "ok, I get it" as quickly as possible, but they'll say it in a really uncertain way that's obvious they don't get it. And then if you ask what part they don't get, they'll say "all of it".

3. Believing they are "too dumb" to do something. Maybe they are dumb, but that's almost never actually the problem. It's almost always laziness. Being dumb doesn't make most things impossible, it just makes them harder. This usually comes after you try to explain something for a while and then they get frustrated and give up and say "ugh, I'm just too dumb for this" (which is also an annoying plea to be told "no you're not too dumb"). In reality they are too lazy and not willing to put in the hard mental work to do it. Smart people know that some things are hard and don't come easily and you have to work at it for a while, and if you aren't getting it, it's usually because you aren't working enough, not because you're too dumb.

4. Thinking they're doing a bad job because they're "too good" for their current position. Every McDonalds cashier thinks they could be head of corporate marketing if someone would just give them a chance. Yes, they suck as a cashier and are totally irresponsible and don't come to work on time and don't take any initiative, but that's just because they're "too good" to be a good cashier. Of course this is just fantasy and self-sabotage.

5. Believing that "the system" is somehow to blame for their lack of success. Most of the underemployed believe that there are phantom hands somehow holding them down, or that society should somehow be different in a way that would benefit them. The reality of the world will always be that a minority get the spoils, and you can either earn your way into that minority or not.

6. Believing there will actually be 7 items when the title says so.

12-16-09 | Belle de Jour

I've been somewhat amused by the Belle de Jour story at The Daily Mail or Times Online

A few things strike me. One is that pretty, intelligent girls seem to be very stupid about using their body for money. First of all, it's clear that these girls have no problem using their body for money, that's what they're doing. So, how should you do that to maximize money with minimal risk and bad consequences? There are many better ways than being a prostitute, which has a large number of negative consequences (risk of murder/abuse, risk of disease, risk of reputation destruction, risk of ruining relationships current & future, risk of messing yourself up psychologically, risk of stalkers / pimps that won't let you quit, etc. etc.). Probably the best way is just to marry rich. Barring that, even just dating rich can be very lucrative. You can make massive amounts of money just cocktail waitressing in the right place, and even more stripping. One problem for Belle of course is that she's not actually very attractive, so she can't get the top stripper dollars that you get in places like Vegas. Of course with a little savvy you can still make decent money on the internet, especially if you have some personality and create a character that people want to follow and you're not just a body; then you can make money from premium cam subscriptions, but perhaps even more so from merchandising and advertising connections.

It always kind of amazes me how little money these girls are willing to do things for. It's such a negative potentially life-damaging move, that you should only accept that if the reward was life changing money, like at least $10k (or more if your life prospects are good). But it's not even close, it's less than $100 for most girls (apparently Belle got a few hundo).

Most disappointingly, the actual blog is just really boring. It's mostly high-fallutin' self-indulgent "observations" and gossip. Not nearly enough real gritty factual story telling of encounters. And the stories of her personal relationships seem to indicate she's just very selfish and manipulative and not nice to men, which makes it all much less interesting.

I think maybe I wrote this before (?) but I had an idea a long time ago to start a Hot Chick Hedge Fund. It's really tragic (from an economic point of view) that there are really hot chicks in places like eastern europe that are not using their hotness for maximum profit. They're doing things like turning tricks for $10 or doing small-market porn. Now, they aren't money machines as is, they aren't ready for the US market. But you could take them, teach them better english, and more importantly teach them a bit of class, how to dress, hot to put on make up, how to flirt and talk to men, etc. Now they're money making machines, they could cocktail in vegas, strip, or just try to marry rich. The Hot Chick Hedge Fund spends a few thousand on each girl and in exchange gets 10% of earning for 10 years or something. It's win-win. Hot chicks are just a physical resource exactly like oil or gold, financial utility is made by transporting them to the market that has the highest prices. This is really basic capitalism.

12-16-09 | Ride Sharing

These things are inherently obvious to the most casual observer.

When someone with a car gives you a ride somewhere you want to go, you should not offer to pay half the gas, you should offer to pay *all* the gas. They are providing the vehicle usage, the depreciation due to mileage accrued, the pro-rated maintenance costs, the insurance, the registration, etc. Those fees easily match the price of the gas, plus they are doing you a favor since you don't have the car on your own. You should offer to pay all the gas and be happy about it. I don't think I've ever in my life seen other people do this correctly. Often the passenger makes no offer at all because they think "you were going there anyway, I don't add any cost". If they're polite they will offer to split the expenses somehow. No.

Asking someone to drive you to the airport is very rude and completely inappropriate. A taxi is very cheap and convenient, or there are plenty of other means. Getting someone to drive you takes at least an hour of their time, which costs $100-$200 or more. But that's the least of it. It ties them into doing something for you at an exact hour, which has an even greater hidden time cost because they can't do anything that will intersect that time period. Plus, if your flight is delayed or something, then their appointment is delayed, which ties them into all the annoyance and vagaries of your flight. Furthermore, if you fuck up and make yourself late or schedule it wrong or something, they inheret all your stress from that. It's an absolutely massive thing to ask of someone. It's pretty inappropriate even if you're completely broke. If you are broke and need a ride to the airport, the more considerate thing would be to ask a rich friend for cab fare (though the rich friend is probably too damn stupid to realize you're doing them a favor and agree).

When cars "let someone in" who's trying to merge or turn in or something, they think they're being so nice, they feel all fucking considerate and superior. Not so, it highly depends on the situation. In general by doing that you are making a decision whether to inconvenience the person who wants to get in, or the person behind you. First of all it's pretty fucking egotistical to decide that you are the person who gets to decide whether the person behind you or the person trying to merge gets your favor. Secondly, more often than not, you should favor the person behind you. For example, if you are on a major arterial and there are lots of cars behind you and some guy is trying to pull in, you should NOT let him in, because doing so severely slows the flow for many people behind you. The utility of each action is multiplied by the number of people affected. Obviously there are cases where you should let someone in, like if they've done a dumb thing and got themselves stuck where they're blocking traffic going the other way; in that case not letting them in is a huge dick move. Like most things in life, the correct action is dictated by consideration of the alternatives and how they affect various people, which is what people never do.

On a related note, this is somewhat more interesting of a topic, I think it's a bad idea to help the traffic flow of people who choose to take a very high congestion route. Let's say there's a certain commute pattern in a city that has very high congestion. In order to reduce it, the city makes it easier for those people to fan out onto alternate routes to commute (by providing travel time information for detours, or doing a little bit of through street connection). This reduces their commute time and most people are happy. I contend that this is a bad thing. You have hurt the traffic flow of the people who were on those alternate routes and avoiding the bad commute path. I contend that you actually want the bad pattern to be very painful, because it encourages people to not do it. Similarly you want the good patterns to be as pleasant as possible, because those people should be rewarded for avoiding the bad pattern. Traffic has a sort of self-regulating property where if you reduce it, more people drive and it gets back to equal badness again. I contend you generally don't want to do anything for it, and especially you don't want to spread it out onto alternate routes. The thing I'd like to see more is bypasses and better local routes so that people who are avoiding the bad traffic paths aren't affected by them.

12-15-09 | Gifts

I've always struggled with buying gifts for people, and particularly for girls. Obviously buying gifts for little kids is easy and fun, because they have no money and aren't picky, and also because their parents buy them all the essentials in life, so you can buy them just frivolous things. Also I love that little kids don't get all the pressure and obligations of adult gift giving, and I despise it when the parents force them on the kid. I love it when you see a little kid gets a sweater for christmas or something and the kid is like "aww I wanted power rangers" and the mom is like "now say thank you". Fuck that, don't say thank you, it's a shitty gift for a kid, be honest.

I love getting gifts myself, but I almost never actually like the *object* that the gift is. I don't like decorative objects cluttering my spaces, I don't like electronics or gadgets, I don't like unnecessary kitchen tools. 99% of the time if I actually wanted some item, I would have bought it for myself already. But I still love gifts that show the intent of the gift-giver, something that shows time and thought. I can cherish the gift and then immediately throw the object in the trash, that doesn't diminish the value of the gift.

The thing that makes me feel so awkward giving gifts is the impression that the actual object has to be treasured. I hate it when you buy someone something, and then they feel like they have to use it all the time when you're around to show how much they appreciate it. Fuck that, just do what you want, the usefulness of the object is not the point.

In the past I've generally bought major items for girls that they need, functional objects, often very valuable ones, but functional objects like bicycles, laptops, cameras, furniture, etc. are not what they really want, especially not from their lover, they're disappointed, and it's a missed opportunity for wooing. They want stereotypical girly gifts like jewelry, dresses, all that shit, but I just can't bring myself to buy that shit because A) my taste in it sucks so I'm sure to buy bad things, and B) it's just such an overpriced ripoff that I'm opposed to that entire market segment and don't want to support it. Back when I was young I used to buy girls expensive jewelry and dresses and whatnot, and they would wear them for me even though they sucked, and I just felt mortified, so embarassed for the both of us.

While I'm on the topic, gift certificates are absolutely disgusting. You embarass yourself and everyone involved when you buy a gift certificate. For one thing, it shows you really don't care about the person at all, it's such an obligatory gift. In fact it's often just a punishment to the person because it forces them to go out to some store they didn't really want to go to. It's also just offensive in a capitalistic market sense because they are a pure destruction of utility. They take money and turn it into something less valuable. You don't just give cash to someone because it's crass and thoughtless, but really a gift certificate is much more crass and much more thoughtless.

You can't ever buy anyone clothes, even if you have great taste clothing is all about the fit on the person. Back when I was young, I bought girls lingerie; that's a big mistake too. It sounds like a good idea, it sounds sexy and romantic, you buy them lingerie, they put it on for you, and then it fits weird and you both feel awkard and embarassed, you're trying to pretend that they look great but you can't stop thinking about how the fabric is pinching or bunching all wrong.

12-11-09 | Egged

My house was egged last night. Or I should say my front door was egged as that appears to be the only spot hit.

It's perplexing because it feels like it was a targetted strike, not random kids being hooligans. I did a little walk around the neighborhood and I didn't see any other egged houses; if it was random kids I would have expected to see a whole dozen eggs used up in a spree down the street.

Also my house has a big fence in front of it with a gate you have to open. There's no way you could hit the front door over the fence, somebody specifically opened the gate and came in and threw the egg. That's not the kind of thing random kids on a spree do, they go for the easiest targets.

So now I'm trying to puzzle out who has a grudge against me that would do this (and knows my address). All I can think of is Old Lady Who Ran Into Me. Maybe that's how you settled scores back in the 50's.

The funny thing is this is exactly the kind of thing I often think about doing to people who have wronged me (like landlords, neighbors, and so on). I wish the person had left a note.

12-10-09 | Hate mail

Gmail spam woes continue :

Gmail classifies my own outgoing mail that gets a failure notice as spam. Urg.

Gmail classifies mails that are autogenerated by Google Groups as spam (signups / failures /etc). Good job.

A hate letter :

Dear Toyota Avalon rental car,

I hate you. I hate you when you beep when I unlock the doors, when you beep when I don't put on my seat belt. I hate your automatic headlights and automatic dash lights, that adjust too late and then stay on too long every time I drive through a tunnel. I hate that you're huge and slow and floaty, I hate that you feel like an old American car. I hate that even though you have no acceleration, you will happily spin the front tires, making it even harder to pull out into traffic. I hate that you have fucking push button up/down for temperature and such, which forces me to look at a digital screen to tell what I'm doing; god knows I don't need my eyes on the road. I hate that your volume knob is one of those fake digital knobs that just spins and gives me no tactile feedback on position.

12-09-09 | Possessions

I take very good care of my possessions. I used to be quite destructive of things because I don't like constraints, but I quickly learned that if you're not careful, your shit is in bad shape all the time and doesn't work as well. There's often actually a very big difference between the function of something that's cared for and something that's not. So I make sure my chef's knives never go in the dishwasher or touch other metal or ceramic, I clean and dry my camelback and water bottles every time after use so they never gets funky flavors, I never let any clothing or sponge or towel sit and be wet because that odor never washes out, I rotate my tires and get my oil changes. I lube and clean my bikes and never leave them out in the rain. The amount of labor time and love put into these items far exceeds their value in most cases.

That's all well and good, but someone external can come along and nullify years of work in an instant. It could be an old lady smashing up your well maintained car, or it could be a friend who comes over and chops on a plate with your knife. I enjoy taking care of my things, but it does put me in a bit of a quandary with guests. I find that almost everyone else in the world is so ignorant and/or careless about treating things well that I certainly can't trust them to not mess things up. So I can either be a dick and forbid them use, or I can be a dick and give them a bunch of instructions for how to use it right, or I can be a dick and not say anything but hover over them and wince. This always causes a lot of problems in my relationships.

Anyway it looks like the Prelude might be "totalled" which would be the best possible outcome I guess. I've started glancing at new cars again. I thought maybe I could get a 370Z used now for a decent price. WTF, people are listing the used cars for more than I know I could get a new car for. (for those that don't know, Nissan has standardized "VPP" (vehicle purchase program) pricing that's very low, about $2000 below MSRP, with no negotiation bullshit). I'm seeing used cars for 2000 - 5000 miles listed at right around MSRP. That's *used* for MSRP. WTF, does anybody pay these prices!?

12-07-09 | Food

We've made some exceptional food recently; some reminder notes for myself on how to reproduce things :

Lamb burger with Za'atar, fried egg, caramelized onions, and curried chickpeas. (plate also contains kale, coucous, and favas).


The key thing is the spicing of the lamb burger; put ground lamb in bowl and add lots of crushed garlic and za'atar, maybe a tiny bit of barbecue sauce (*). (never add shit like eggs or bread or onions to burger - this is a burger not a meat loaf, that shit all adds too much water). Stir with your fingers thoroughly, form patties, sear, cook a bit more than you would a beef burger, medium - medium/rare instead of rare - medium/rare. (* = bbq sauce and garlic will caramelize much faster than meat would, so beware of burning the surface, and don't use too much, maybe 1 tsp per pound). Dress with mayo and hot sauce. The curry and onions and fried egg were just about perfect toppings; all the greasy parts and egg yolk and mayo and curry run together and form the most delicious sauce known to man.

Oxtail Chinese/Taiwanese soup bastardization :


also contains soy sauce egg (*) (which is the wrong culture, I know, and then I added thai basil and sriracha flavoring on the table which is another no-no, but delicious), kabocha squash, pickled vegetable.

I did the "Chinese/Hawaiian" thing to do with oxtails, which is to blanche them first in boiling water, rinse them off thoroughly, then brown them in a hot pan, then deglaze and make your soup from there. Supposedly this initial blanche and rinse removes "impurities" and makes the soup have a cleaner beefier taste without some of the more funky flavors you can get from blood and such. I have no idea if that holds any water, but it turned out damn good.

Oxtails take 3-4 hours simmering on low or braising. You basically start the soup just like any other braise - brown the meat thoroughly, add some onion, garlic & ginger, brown it a bit, then deglaze with rice wine or sherry, add some spices (star anice, clove, fennel seed, etc), add water to just barely not cover the meats. Simmer very low or in a 300 degree oven.

When the meat is about to fall off the bone, remove all the oxtails and set aside; once it cools, then take the good meat morsels off the bones. While waiting, pour off the soup and strain it to remove all the solid aromatics. Return the broth to the heat and cook the kabocha chunks (10-15 mins) or whatever else you want at this point (turnips, carrots, bok/ung/pok choy, what have you). Once kabocha is nearly cooked, remove it so it stops cooking, don't leave things in the broth.

Boil noodles and assemble soup bowls on demand from separated parts like they do in asian restuarants.

One thing I would do differently next time is trim some of the fat off the oxtails before cooking. It's too hard to skim out of the soup, and it was just a bit too greasy.

(*) the soy sauce egg was a real joy on its own. A few weeks ago when we went to Vancouver we had some amazing ramen at Benkei, which looks like this : 1 or 2 . They serve a superb soy-sauce egg that's hard boiled with just the perfect glassy gelatinous yolk, then soaked in soy which gives it a salty funky depth. I used roughly this technique to reproduce it.

Boil egg for about 7 minutes. Remove, run cold water on it for a minute or so, then plunge in ice bath. Put ice bath in fridge and let cool completely (about 30 minutes). Once fully cool, peel egg and place in soy marinade (*). Let soak 3-5 hours. Yum. (* = I used just soy sauce and a bit of water for the marinade, but that was a bit too strong; I'm not sure what Benkei uses, but I'm guessing it's got some Dashi in it or something, maybe a bit of sherry would be good too, a little something to balance and cut the soy). I think my egg was in 3 hours and you can see it's only just browned; to get the super brown egg you see in other pictures I guess requires more like 8 hours; so do the prep in the morning to have it for dinner. I think the vinegar in the ice bath is a total red herring.

12-07-09 | Balancing Ranges

When you go to the bathroom, you should always turn on the fan. If you only turn on the fan when you're doing something stinky, you are creating a strong correlation between the external observable (fan on/off) and the internal hidden state (stinky). You can't hide the fan on/off state, so your goal is remove the correlation. You do this just by always turning on the fan.

In poker this is called "balancing your ranges". Basically it means making sure that when you do any given externally observable act, you do that same act with multiple different hands, so that the external observable is not highly correlated to a specific hidden state (your hole cards). For example, say you only ever raise under the gun with AA/KK - we would say your "range is not balanced". One way to fix this is to raise some bluff hands as well. Another way to fix it would be to limp your AA/KK (assuming you limp enough other hands to have balance). If you do limp some hands UTG but then you only reraise with AA/KK , you're unbalanced again, so if you are limping AA UTG then you also need to limp-reraise bluff from UTG sometimes to balance your range.

12-07-09 | eBay

eBay is such a fucking scam. I recently sold a bike for $1280. eBay takes a total fee of $22.35 ; PayPal (also eBay) takes a fee of $37.42 ; UPS charges $125 ($80 shipping plus the fucking +$45 large item surchage scam).

The net result is that people pass items back and forth, and eBay and UPS get rich.

I thought the UPS home pickup was pretty sweet. For those who aren't aware, you can request a home pickup for free any time; because they do residential in the evening usually, you can request a pickup even after 3 PM and still get it the same day. You print your own label and out goes your box.

I was not aware of the large item scam. UPS apparently automatically remeasures your box in transit. This is sort of a scam because if they smash the box and it gets bigger, they will remeasure it and charge you. The large item scam is this sudden extra charge that they tack on in a weird way. If the

Length + 2 * ( Width + Height ) >= 130 inches

A standard size bike box is 54 x 8 x 30 , so you're at 54 + 2 * ( 8 + 30 ) = 130 exactly. If they remeasure it and the box is bulging at all, like if the width is now 8.5 inches, boom you get the +$45 charge.

Hey fuck you, we have a monopoly, what are you gonna do about it? Not use UPS? LOL. What can brown do for you? We shit directly in the customer's mouth and they come back for more.

12-07-09 | Out of Commission

I got in a car accident on Saturday. N and I were driving to buy board games and look at the Olympics from the shore when Old Lady ran a red light and smashed into the passenger side of the car full speed.

We were stopped at a red light at an intersection, our light went green. The other car in the lane to the left of me took off, and then I more slowly went into the intersection. Just then I noticed a car to my right coming down the street at me awfully fast. This was the worst moment of the whole thing - for about one whole second there I realized that the car coming at us was not stopping, and there was nothing I could do about it. I think I said "whoah" or something which N heard. I had enough time to think "should I brake or accelerate?" and then *BAM* it smacked into us, right into the side of N's door.

We were pushed sideways into the car in the left lane, which smashed in the left side of me a bit. It seems neither N nor I are hurt, so it's no big deal. (please refrain from all "are you ok?" or "I'm glad you're ok" emails). It is a pain in the ass because the Prelude is a bit too damaged to be driveable, so now I have to get it fixed or maybe get it totalled, blah blah fucking life is such a hassle.

Picture of Smashed Prelude

One thing I realized after the fact was that the accident was so quiet. There was no squealing of brakes, no swerving out of the way. Old Lady didn't slow down at all. Normally in an accident like that you expect a last minute braking, some tire tracks, all that. Nothing. When we got out of the cars to talk after the accident she was in shock of course, I was trying to get her to say what she was thinking and she mentioned how the light was green for her. That's scary, she just completely didn't see the world for many seconds there.

A few things trouble me intellectually about the whole situation.

1. the insurance will not come close to restoring the value lost of my car. Even if they fix it "completely" it has another accident on its record which greatly reduces resale value, and god knows if the frame is slightly bent they never really fix that right. I believe technically that insurance is supposed to compensate you for lost resale value, but they never do. So even when insurance is paying out, they screw you. Also, since Old Lady is completely at fault, the compensation should be way beyond just fixing the car, there should be a penalty for all my distress and lost time.

2. I've become hyper aware of how vulnerable I am and how you have to trust the other cars. Walking across streets since the accident I keep looking at the cars coming at me thinking "he has a red light, but who knows if he's actually going to stop? he might be old, or on a cell phone, or getting a blow job, and just run right through the red and smack into me". It's terrifying.

3. Old Lady and her complete unfitness to drive is very disturbing. Our roads are plagued by old folks and other unfit drivers who should absolutely have their licenses taken away, but both the state and their families are too fucking pussy to do it. My own family and myself have failed to stop our grandparents from driving even though they are horrible dangers to themselves and others. Basically just because we're afraid of the big scene where they get mad at us. It's very sad to confront your parents about their failing abilities in their old age, but you have to do it.

4. My neck is in rather a lot of pain since the accident, I'm sure I have some kind of minor whiplash. I almost certainly won't do anything about it to get compensation, but it occurs to me that other people might. By not doing anything about it, I'm basically just costing myself money, because there's some random chance that I hit someone some day and they do claim medical damages from me. It means my net EV for collisions is negative, not neutral.

5. I have to decide whether to file through my insurance or Old Lady's insurance. Under my insurance I'm limitted by the restrictions of my policy (which is a super-minimal policy because I despise insurance). On the other hand, if I file through her insurance I'm sure they'll try to fuck me as much as possible.

6. Thank god that Old Lady was driving a Honda Civic. Because of that, her total momentum was reasonably low, even at 35 mph with no braking, and perhaps most importantly, her bumper was at the right height for the Prelude's impact absorption pillars, and her front end crumples. If she had been driving a Suburban or something like that, my girlfriend might be dead right now.

12-04-09 | Sexual Bullshit

I'm so sick of reading the liberal codescending media about embracing "kinks" and the idea that any sexual behavior is okay if you're open about it with your partner. Bullshit.

First of all, all sorts of weird sex acts with your partner are wonderful. Go nuts. That's basically never what "kink" is about. Kink is really not about sex at all. It's about creating weird little role plays and fixed scenarios that satisfy some emotional defect in your fucked up little brain. People in S&M think they are "a top". No you're not. You're a fucking loser in real life who was maybe beaten as a child or some shit and you feel out of control in the real world, so you want to create a phoney scenario where you can be in charge and dominate someone else. It's not a "healthy kink" it's a fucking mental derangement. Granted it's a pretty harmless derangement, unlike molesting children or voting Republican, but it's not a healthy alternative behavior. Similarly with "furries" , etc. etc. - being a furry is not a healthy alternative sexual lifestyle, it's fucking deranged and clearly indicates serious psychological issues that should really be worked on, not accepted. (in this discussion there's a big distinction between people who just *play* at being a top/bottom/furry/scatologist/necrophiliac/whatever vs. identifying that way as a major part of their personality and/or only being able to have sex in that character). (you could say similar things about Japanese guys who are into Bukkake or rape simulations ; no that's not a "deviant" or "alternative" sexual taste, it's fucking sick and reflective of distortions in society which is making their minds so broken).

Similarly with "polyamory" ; the modern permissives like Dan Savage like to spout this bullshit about how having many simultaneous long term sex partners is perfectly fine as long as you're honest with your partners. Yes, it's certainly better than cheating, but let's not kid ourselves. Someone who identifies as long-term polyamorous is a fucking sicko (if you've actually met any, you know they are always *creepy*, like they have those googly eyes and they stand way too close and touch too much), and it's generally manipulative and abusive of their partners. The partners don't want their lover to be polyamorous, they feel like they have to agree to it or lose them; someone in a long term relationship will develop emotional ties and hopes. It's indulging in personal pleasure at other's expense, which is reprehensible. Dan and others perpetuate this liberal claptrap that the other person should "just say so" if they don't like it. That's ridiculous, just because someone doesn't speak up doesn't mean the abuse is okay.

That's not to say that I think these things are horrible. We're all human and have our flaws and weaknesses. If you have a derangement and you can channel it into a behavior that satisfies your insanity and doesn't hurt others, good job, that's better than a lot of "normal" behavior like the guys who go out and get drunk and pick fights in bars, or the girls who go out and get drunk and fuck a random guy, or the parents who repress and yell at their kids, all of which are standard good american bible belt practices and are much more destructive than being a cross dresser or whatever west coast "kink". But let's call a spade a spade - it's weird.

12-04-09 | You all suck

Maurice Clemmons apparently killed a few cops up here. On Nov 30, he was a suspect with a manhunt on to find him. There were no witnesses, only weak circumstancial evidence that he was the killer. He was pulled over in a traffic stop and was shot and killed by an officer who saw no gun. It's hard to say whether or not the shooting was justified or not, but it's definitely questionable. For one thing, the officer only identified the person he shot by visually seeing that he "matched the description" of Clemmons - this was in the middle of the night in darkness. Without knowing that the person was in fact Clemmons, or that Clemmons definitely was the cop killer, and without seeing a gun, the cop killed this man. Luckily the man shot actually was Clemmons and he was in fact the cop killer. However, I made the mistake of reading the comments on the news story :

seattle times or seattle weekly

Here's a semi-random sampling of the comments :

Good job, coppers!

kiro7 twitter says he's dead. I'm glad this saga is over and we don't
have to see him in court. 

Now lets hope they shot Clemmons just like the other recent police
officer killer; paralysis for life. Only Morfort was paralyzed from the
waist down. In an indeal world this S.O.B. would be a quadraplegic. 

The perfect ending for a cop killer: capital punishment by cop.

Well, best way to get him. I don't want to pay tax after any of the
criminal just to have theire "house, food and amusement" in jail. Good
job officer :)

yeeah kill that sucker I am glad he is dead, do not let this monster live just kill it

It's pretty amazing how completely retarded you all are. There's absolutely no concept of the rule of law or the function of courts, there seems to be no concept of the idea that cops assasinating suspects might be a bad thing. I'm just disgusted.

I see the same thing with the Republican comments about Guantanamo. They seem to think it's perfectly reasonable to torture *suspects* and give them no access to courts, no human rights. Now, it's one thing to debate torturing someone who you *know* is a terrorist, but these people are suspects and we know for a fact that over half of them are completely innocent. We've seen many cases of people who have zero connection to any terrorist cell being apprehended and extradited, and you still think it's perfectly reasonable !?

I listen to NPR on my commute and it almost always makes me furious. (for one thing the fucking Steve Scher who does the call in show here is such a dull moron, I just want PRI and Pacifica and BBC please). He had a guy on the other day talking about what Mike McGinn might do to make Seattle a better place for bicycles (which was a big part of McGinn's campaign). The guest was actually really smart, he made some excellent points that I hadn't really thought of before. One is that the difficult and dangerous bike routing that we currently have in Seattle causes a population selection - only young, bold, fearless, dedicated people will ride; the older more reasonable safe people won't ride because it's too dangerous and the bike routing sucks too much. What that means is the biker population is biased towards kids who antagonize cars and it creates a vicious cycle. In cities like Copenhagen or Portland where you have lots of good bike routing, that encourages a wider spectrum of the population to get out, which makes the average cyclist more calm and considerate, and gets more people interested in supporting good bike routes, which is a nice positive feedback cycle.

Anyhoo, I'm feeling happy listening to this smart guy and then they start taking calls. Every single call is "the real problem is the cyclists don't obey the laws". The call in guy is like "okay, do you have any comment on public policy?". Next caller is "I drive and all the time I see cyclists just cruising through stop signs". Okay, do you have anything useful to say? Do you really think that if the cyclists stop at the stop signs they are going to create such wonderful goodwill that the city will shower them with money for better bike lanes? Are you fucking retarded? Can you vent your spleen on your blog and not the fucking radio?

Today I'm listening and they have Obama on talking about the economy. He says jobs only declined slightly last month. Big applause. He says he is encouraging the banks to lend more to main street. Big applause. Seriously? Are you so seduced by his serpent's tongue? He has done *nothing* for jobs, and *nothing* to reform the banks. He's absolutely in their pocket. His team seems to buy the ridiculous macro-economic mumbo-jumbo that increasing GDP is all that matters, even if that GDP is nothing but financial industry profit.

12-04-09 | Bike Optimization

I mentioned before about how saving money on cheap inner tubes is dumb. So there's this amusing site Weight Weenies with lists of lots of bike components and weight.

As soon as I saw it I immediately thought "MMO". It's just like the list of gear for kitting out your character. Should I get the ring of +1 str or the bracers of +5 HP ?

It's funny to me that people seem to love doing that sort of optimization in games, but not in real life. People love MMO's with lots of gear, they love to sit and think about the best way to kit out their character and how to maximize his stats and all that. It's a very detailed cost/benefit analysis, and lots of people do it. But those same people either suck at it miserably in real life, or even just refuse to do it, saying something like "I don't want to live that way" ( "that way" being rationally and logically ). So you don't want to think about maximizing benefit/cost in real life, but you enjoy it so much that you choose to spend your leisure time doing that activity, hmm. Maybe if you would think about benefit/cost in real life a bit you would decide not to spend your time playing an MMO. Anyhoo...

Weight Weenies obviously presents the idea of computing the weight of each part of a bike and its cost and finding the optimal weight reduction strategy for a given budget.

A lot of people would approach this with the "greedy strategy". That is, first find the one item that has the very best (benefit/cost) slope and buy that upgrade. Repeat until you are out of money.

This is obviously wrong for various reasons. For one thing, the upgrades are not additive, an upgrade to the same "slot" (eg. the "ring" in an MMO character, or the "seat post" on a bike) replaces the previous item in that slot. That means you only ever want to buy one item for each slot, and the greedy algorithm does not guarantee that you buy the best item for a given slot. The greedy algorithm also doesn't deal with using the finite budget well, it could leave you with left over money that isn't used well.

The obvious solution is a kind of dynamic programming like the LZSS optimal parse. Consider all prices to be in integer dollars. You have a budget of N dollars, think of that as the "X" axis on the time line. You are trying to walk from X=0 ($0 spent) to X=N (full budget spent) and get the maximum step on the Y axis (benefit).

So you can make this a graph. At (X=0,Y=0) you have the initial node with all M upgrade paths coming out of it; each path has an X length equal to its cost and Y length equal to its benefit. After you take a path, all upgrades on the same slot are excluded, so the next node has a reduced set of paths. Note that the graph does *not* branch exponentially, because the nodes come back together. That is, an upgrade like {ring,bracer} is the same as {bracer,ring} so they go to the same node. If you had infinite budget, the paths would wind up all ending in points with no exit paths (there would be many of them, equal to the number of possible ways to do a full-slot kit-out).

You can phrase this another way to make the dynamic programming aspect obvious. (dynamic program just means caching shared bits of a computation). If you have D dollars remaining and you have the set S of upgrade slots unused, the item you choose next is I(D,S). I(0,S) is null. I(1,S) is the best item in the set S that you can get for 1 dollar. Then :

I(D,S) = arg_max(i) {  Benefit(i) + Benefit( I( D - cost(i) , S - type(i) ) ) }

i = an item with type(i) in the set S and cost(i) <= D

arg_max = return the iterator which maximizes the enclosed expression

One way to solve it is the LZSS way which avoids any recursion : you start at the "end" of your spending, when budget remaining is zero, and you walk backwards doing the above computation for each D, until you reach D = N (your initial budget), and then I(N,all) is the first item you should buy (and you follow the paths back forward to get the full list of items to buy).

The other way involves heavy recursion. You just ask for I(N,all) , which will call I() on lots of sub choices, which calls I() some more until you have your answer.

I think that's optimal;

12-03-09 | Web

Jeebus the new Google fade in is fucking retarded. Seriously fucking quit it. What kind of fucking ridiculous stupid fucking retard thought of that.

"I know, let's hide all the buttons that users want to click, people will love that".

"You know what would make this page great? If it took longer to load and offered no more features."

You're fired.

It's really annoying the way I can't "own" the web. I can't freeze web pages at versions that I like. People keep changing shit under me just when I get it worked out nice. My banks and credit cards and such keep making their pages worse and worse, adding more Flash and fucking animations and gizmos and shit that just make it so much harder to find things. The fucking Chase pages have some kind of adaptive menu system where they never show you all the sub-pages in one place so you have to navigate a literal labrynth of links to find the page you want (ala Zork). Even that would be okay if it just stayed the same because I could memorize the steps, but every few months they change it so I have to go hunting again.

12-03-09 | Following

The modern emo metrosexual dorky man is often afraid to take charge and lead his lover. He's afraid of stepping on her toes, making her do something she doesn't want to, afraid of suggesting something that doesn't turn out well and then getting the blame for it. It turns every discussion about what to do into this huge sissy yielding back and forth, like "what do you want to do?" "oh I don't know, what do you want?" "well I'm flexible" "well I could maybe watch a movie" "yeah you want that? I could do that" , ugh it's revolting.

He'll claim that he's "sweet" or "considerate". Really it's nothing but cowardice and weakness. He doesn't want to just boldly lead where he wants to go, because he's afraid of rejection, afraid of the blowup if he does the wrong thing. The woman wants you to lead - so yielding and refusing to lead is not doing her any favors.

One way to get over this is of course just to not care, aka "be an asshole". BTW this is why it often seems like girls are attracted to dicks and assholes - because in many ways they have positive traits that the sensitive man lacks. They don't actually want an asshole, and guys who say "women like jerks" are being foolish and defeatest. Obviously they want someone nice and considerate, but considerate does not mean pansy. It means thinking about the affect on them when you lead.

A crucial point is that the art of Following is just as important as the Leading. It's like ballroom dance. The follower needs to subtly direct the leader, and follow smoothly, taking the cues and then moving on their own.

Perhaps the most important thing is that the follower needs to speak up immediately when the leader takes them in a direction they don't agree with. This gives the leader confidence to know that he can lead and not have to worry about "is this okay? is this okay?" all the time - he knows the follower will speak up if something is wrong. Often when I'm a guest in somebody else's house, I find myself really uncomfortable, and a large part of it is because I feel like they're afraid to just say something when I do something they don't like. This makes me hesitant about every action because I don't know what will displease them. If I knew they would just say "hey don't put your feet on the coffee table" , then I could be at ease and just do whatever.

Also, the follower needs to reward the leader when he does a good job, with enthusiasm and good spirits. If you take someone on some fun adventure and they don't go along with it and don't appreciate it, you won't want to do it again.

Almost everything in human perception is "relative to recent average". That means when you improve your situation, you rapidly acclimatize to it, and find all new things to complain about.

It's also something important to be aware of if you are trying to maximize how other people perceive your value.

If you are just nice to people all the time, they won't think of it as being nice. For example, say you just do the dishes every time your wife asks. She won't think of it as you doing something really nice for her, she'll just think "my husband does the dishes for me, that's the way life is". Instead, if you refuse and moan about it usually, but *ocassionally* do the dishes, when you do it she will think "what a sweetie".

Similarly, at work if you just put in solid hours and do good work all the time, you will become invisible to your boss. Instead, if you generally screw around and chit-chat a lot, but *occasionally* you really crunch and do some amazing work, your boss will think "oh, that guy really gave his all for the team there" and you'll get big bonuses.

I think there's a very fine moral line between "manipulation" and just being aware of how people perceive your actions.

If you find yourself being too consistently good, you should intentionally fuck up. Like say you've really sexed up your lady really nicely a few times in a row - you better do a terrible job on your next go, or it's going to become the "norm". Once it becomes the norm, then any deviation downward is seen as bad, and continued good jobs are not perceived as good, just as normal. You need to fuck up often enough that the norm is perceived to be low enough that you can beat it.

When you do something nice for someone, you're usually doing it to be *perceived* as doing something nice, not actually doing something nice. When you bake cookies and give them to someone, it's not because you think they really need a cookie and you're helping them out, it's because you want them to think "oh how nice".

Acknowledging this, when you do something nice, it should be the thing that creates the maximum perception of nice while costing you the minimum.

For example, when I do something for someone that takes an hour of my time, that costs me $200 or whatever my hourly time value is. They certainly do not appreciate that for its fair value. On the other hand, I can do something trivial like buy someone flowers and it's perceived as very nice.

Often I will choose to do the things that are "actually nice" as in actually helpful to the person, even if they don't particularly perceive it as very nice, and I refuse to do the the easy things that have high niceness perception because they're "cheating" or "too easy". Really I'm just being stupid.

In the world, we can see our choices, and we can see the results they will produce. There's no omniscient judge who is watching your choices and giving you gold stars for being "good". There are only the real world results. You should make the choice that provides the best real world results. That means buying flowers, cheating on your taxes, bribing restaurant hosts, making smalltalk, treating women like dogs, driving in the carpool lane, etc. I can't quite make myself do these things, but I also don't understand what exactly is holding me back.

12-02-09 | Board Gaming Ethics

0. Be ready to move on your turn and keep the moves fast. Don't chit-chat during your turn, and use the time when it's not your turn to plan your actions. Go to the bathroom or whatever immediately after your turn. In general don't make people wait for you. Play fast.

1. Don't quit when you're losing. Don't say "this is stupid" or complain about your "bad luck" all the time. Hey guess what, board games are probabilistic and have high variance. Usually when you win or lose it's luck, the skill shows up in how often you win or lose. Don't pout and make it obvious you can't wait for it to be over, stay interested in what the other people are doing.

2. Don't drag out runaways when you're winning. When it's obvious you will win and the loser has no hope, it's polite to offer them resignation rather than dragging it out.

3. Always be trying to win. Be competitive. Games are no fun for your partner if you're not trying. Also, it's no fun for everyone if you play your own "meta game" and make up your own goal (like defeating some other particular player in a multiplayer game, or getting the lowest score possible, etc.).

4. Don't make deals or alliances outside the spirit of the game. Also deals and alliances should only be made that are in your best interest, not to intentionally screw someone else (unless that someone else is winning and it's in your best interest to screw them).

5. Don't exploit badly broken rule systems or loop-holes. Try to play within the spirit of the game. It's polite to say "I think I could do this move, but it's totally not fair and breaks the game, so I'm not going to". If you're aware of a rule that others aren't, don't keep it secret and spring it on them, warn them about it. If someone is obviously making a move because they don't know the rules well enough, tell them about it. In general, do not treat knowledge of the minutia of the rules as an aspect of skill and a way to win.

6. Tally your own accounting. Be aware of when you need to pay or collect resources and speak up and do it yourself. Don't rely on others to see it for you. It's good ethics to alert someone a few times, but after that you are under no obligation to do so and should just keep playing fast if they consistently fail to pay attention.

11-30-09 | Some more bike notes

There really is a *huge* seasonal variation in the used bicycle market. I knew this, but I thought it was a small factor; no, it's massive, particularly in the low end. There are just tons of cheap old bikes for sale right now for under $200. I have one I'm trying to offload and I've got zero interest. You just cannot sell old bikes right now. Conversely in the summer when I was looking at buying bikes, they were few and far between. If you ever have any ability to time your bike purchasing, do it. Also, bike shops are slammed in the summer and often low on stock. They will not give you good service and if they sell you anything it will often not be what you want. Much better to go in the winter, and they will very likely negotiate quite a bit on price with you in the winter.

I just shortened chain on my new bike again. Turns out I put it on way too loose. I think particularly with a compact double (because there's a big difference in ring sizes from 50-34 vs. 53-39), you really need the chain to be absolutely as short as possible. That means big ring to big ring plus one link. You really do not need any slack for the derailleur at all - the derailleur can move to the chain. And you will never really ride big-to-big. The problem I was having was chain slip when I was near the little-to-little rings. I now know this was because I had too much slack in the chain (riding little-to-little on a compact double with a pretty big cassette range isn't a great idea in any case).

Some tips on buying a good old used bike : I think the best bargains are actually in pretty recent low end bikes; 2-5 years old, generally aluminum, they just have zero sex appeal so there's no markup. You can get used road bikes from chinese brands like Giant or the Performance Bike generic brands with Tiagra level components for $200 and they will work better and be lighter than very expensive old bikes. If you want an older sexy steel bike, here are a few tips :

1. In general you want to look at bikes from about 1985 - 1995 ; after 1990 would be preferred. If you get a bike that's Shimano components post about 1985 , that will give you compatibility with the whole modern generation of components - all the recent generations of 8 & 9 speed pretty much work back to that era (new 10-speed stuff doesn't). Avoid old Dura Ace stuff, though you probably won't see it. Avoid the older "ten speed" era bikes. Avoid French bikes at all cost because they have weird and annoying compatibility problems that make parts much harder to find and more expensive. Avoid old Campagnolo unless everything on it works perfectly.

2. I think Japanese bikes like Bridgestone and Miyata are a good way to get quality stuff without paying the "sex appeal" markup of Italian bikes. Brand names and collector's items will have big markups that you don't need to pay, you can find great stuff that's just as cool if you do a bit of research.

3. Make sure the frame is a more modern higher quality type of steel that doesn't weigh a million pounds. Obviously if you can get a real sexy frame that's like Reynolds 531 or 853 or one of the better Columbus or Tange steels, that's nice. Avoid the pre-1985 straight steel super heavy shit that's in bikes like Schwinns, Pinnacle, Peugot, etc. Look for "butted" on the stickers. Don't get sold on paying a premium just because of a "Columbus" (or whatever) sticker though, those are just steel tubing brand names and they made a whole line of steels, some cheap, some premium. But really don't focus on this other than avoiding the super old thick crazy heavy shit. If the frame is good, make sure it fits you nicely, and then you can always replace components.

4. Look for rust, dents, big gouges. You don't want to deal with any of these. A tiny bit of surface rust on one or two spots of the frame is not a big deal, usually you can just scrape it off.

5. Check for parts being frozen. Take your tools with you. In particular check the stem is not frozen in the steerer and the seat post is not frozen in the seat tube. Take off the pedals and make sure they are not frozen. Again it's not worth buying a bike that has these problems. Often people who owned these old bikes never greased them or serviced them a single time in the last 20 years.

6. Bikes that are badly adjusted can be a nice bargain. They will seem to brake terribly and shift badly, etc. The owner thinks it's "in bad shape" but you can tell it's structurally fine. Usually when I buy an old bike I plan to throw away the tires and tubes, brake pads, and cables. Replace them all it, clean it, lube it, tune it, and it works like a dream. That overhaul will cost you about $100 in parts.

7. Bikes sold with "extras" can be a big bargain. You'll see naked bikes for $150, and then you'll see bikes with fenders, racks, bags, lights, etc. for $250 ; those parts alone are worth $250 if they're in good shape. Sometimes just one component on a bike can be worth the whole value when people don't know what they have; something like a premium saddle is worth $100+ and you can find them on $200 bikes.

8. I recommend looking for a frame that's versatile. Look for braze on fender mount eyelets at the fork and rear dropouts. Old touring frames are great to turn into commuters because they have the good rack-mount brazeons on the seat stays. Look for brakes with clearance and space for fatter tires and fenders. You might never use any of these things, but it's good to have the flexibility so that you can do whatever you want.

Don't try to save money on inner tubes. The difference between a cheap tube and a good won is actually quite large. A cheap tube is around 125 grams, a good one around 75 grams. 50 grams may not seem like a lot, but shaving 100 grams = a quarter pound off a bike would cost your $50 - $100 , so saving $5 on tubes is highly irrational.

Tires are even more important than tubes. The same is true of cars of course, I've written before about how retarded it is when people buy a $50k sports car and then put cheap tires on it. It's just a radically bad price-performance decision. Of course those people don't actually care about performance, they only claim to; in reality they are spending for image, so maybe it is a logical price/image decision.

More on this later.

11-25-09 | Blurg

mscorsvw.exe gobbles CPU to "optimize" your .NET CLR assemblies. It's gotten to the point that computers are almost always running some kind of "background optimization task"

I stuck some egrips on my Canon S90 and I'm pretty happy with the result.

I updated my picture slideshow screensaver cbsaver with a "ForceWindowed" option, and a special mode if you set TransitionMillis to 0. ForceWindowed makes it use a full screen size window instead of actually going to full screen ddraw. This takes a lot more CPU, because it uses blits instead of flips, but I found the full screen ddraw caused some odd flickers on certain graphics cards. The whole point of cbsaver is that the transition from desktop to screen saving should be completely smooth and non-distracting, so the screen jumping to full screen was unacceptable. The TransitionMillis=0 mode makes it just flip pictures and skips a lot of the complication so it's much faster; the goal of this is to avoid spinning up my laptop when it's in low-heat mode.

Which reminds me : I'd like to guarantee that cbsaver in TransitionMillis=0 mode doesn't make the CPU come out of low speed mode. If I use a lot of CPU for some reason, I'd like my app to just run slower, not kick up the CPU clock rate. So far as I know there's no way to do this. I have some hacks in there to try to avoid it, for example I don't flip to an image and then immediately load the next one, I flip and then wait a few frames to do the load so that I spread out my higher CPU use tasks over time. (really I should have the load on another thread at low priority).

11-24-09 | Google Voice

I'm now using Google Voice cuz it's better than nothing, but it has some major usability problems :

1. Not integrated with Mail and Reader. I have to keep 3 web pages open if I want to see all my junk. Yeah yeah I know can make an iGoogle page or some shit to put them together but I shouldn't have to do anything. Deliver me all my notifications together please.

2. No realtime refresh, no sound notification on new messages. If I want to see if I got new messages I have to go over and hit refresh !? Not cool. I should be able to tray it. Really it needs to be a proper Win32 app and behave more like Trillian or other IM clients, not some web shit.

3. It collapses threads like a motherfucker. That's totally inappropriate for SMS conversations where each message is 1 sentence. It starts collapsing at 5 messages, which means pretty much every conversation is collapsed all the time. I find myself just constantly clicking the "N more messages" buttons to make it fucking un-collapse. Also it seems to have no sensitivity in the collapsing to how recent the thread is. Obviously the fucking thread that I'm writing in RIGHT NOW shouldn't be collapsed, but the one from two weeks ago could be collapsed even a lot more.

4. Disabling ring-through is buried too deep in the settings. It should be a toggle right on the main page whether to ring & forward through or not. My main usage now is when I sit down at my desk at work I want to disable all the ring-through and SMS forwarding, and do all my "phone" at my computer. When I get up again I want ring-through to be re-enabled. (Also if I forget to re-enable it, it needs to be easier to toggle from my phone!).

5. Previous complaint about SMS forwarding not setting caller id still stands. Hopefully this will get fixed, but I'm becoming more tolerant of it.

11-24-09 | Feedback

One of the hard things about data compression is that when you first try an idea, chances are it will appear to be a loss. That is, say you have some working compressor like an LZ77 compressor and you try some new idea like using a reduced set of possible offsets instead of all possible offsets. When you first try this, you won't have fully worked out the idea, so your implementation is not that great, you don't yet realize the key to making it profitable, so it will appear to just hurt performance.

If you have faith in the idea and stick to it and work out the fact that you need to context-select the reduced set, and only do it when the prediction set is good, and escape out to context coded literals, etc. etc. all the subtleties, then it becomes a win and a nice technique. But for a long while there you aren't sure. You feel like you might be headed down a dead end, wasting a lot of time refining a technique that will wind up being worse than the original (of course this has happened to many people many times with techniques like LZ78 and DMC and Fractal images, etc.).

This is really the key skill that makes the practice of science an "art", and what distinguishes good practitioners from muddlers - it's the sixth sense for what is a good way to spend your time, what areas to explore are likely to be profitable, and how long to stick with it before you punt. If you punt too soon you might miss a huge discovery, but if you stick with it too long you waste your life chasing dead ends.

Of course this is a common feature of all the arcane arts. Any field where you get linear feedback to your exploits is trivial (*). That is, if you start down the right path and immediately get good feedback that it is the right path - anybody can do that, it's easy. All the difficult fields share the property of nonlinear feedback - including life of course. For example I believe that "dieting" and "working out" are really fucking easy life tasks because they provide very obvious linear feedback - you act and you see results. Harder things are things like choosing to make a big career change, since it will get much worse before it gets better, or choosing to work on a failing relationship.

* = obviously that's not true, there are "simple" fields that give linear feedback that are still quite difficult simply because they are a slog. I think Chess for example is one of them; if you spend more time studying, you get better results, pretty linearly; Chess is still difficult because the sheer mental focus and rigor to put in enough study is exhausting. Poker on the other hand is not simple because of the very large variance; you might adjust your game in a good way and immediately have some very bad results due to bad luck; you have to have intuition and faith to know it was a good adjustment even though the results don't seem to indicate such.

11-20-09 | Bike Tires

A lot of people believe that they should get bike tires with a bit of tread pattern, grooves or knobs or something for "good grip" or "water channels". This is a lot of crap that's gotten into people's heads from the car tire advertisements. See the experts : 1 , 2 , 3 on tire choice.

So now that you've read them you should be drinking the kool-aid that slick tires are the way to go because they maximize grip on pavement and will never hydroplane.

I don't think so. The truth is, grip on smooth pavement will never be a problem with bike tires. As Sheldon's chart shows, it's basically impossible for you to ride fast enough or corner hard enough to exceed the friction limit of a bike tire on pavement - even wet pavement. So why optimize for that? Slicks might give you optimum grip, but you already have plenty.

In reality, grip issues when biking will occur when your tires *aren't* in contact with pavement. I've had bad wipeouts due to gravel, sand, etc. In Seattle we also have the "steed" - the rotting leave gunk that's as slippery as a banana peel. When your tires are on steed you have no grip, and a bit of a knob to cut through that and get to actual pavement would be a huge boon. (it's called "steed" because rotting leaves are almost the most disgusting thing in the world, they're goopy and brown and smelly like a giant film of shit that gets on everything - second only to Paul Steed who is the most disgusting thing in the world).

I don't know what the best compromise is for a tire that can cut through some steed but still have good pavement grip. Clearly if all you care about is wet pavement grip, the right thing is a slick tire that's as wide as you can fit, and underinflate it very slightly.

11-19-09 | Pathetic

There's nothing worse than being intelligent but lazy. Intelligent enough to know that all ordinary activities are banal and pointless, but too lazy to do anything you think is actually interesting. It's a pathetic state of feeling superior but being despicable.

Dined at Tilth the other day. I won't even write a review, but I need to stop going to "New American" restaurants because they're just so fucking boring. Seared protein, buttered veg, some broth or foam, SNORE.

The best season of The Simpsons is Season 5. Season 4 (of Mr. Plow - though the best episode is Homer the Heretic) is a little bit too rough still (earlier seasons are way too rough). Season 6 (of Itchy & Scratchy Land and A Star is Burns) is really good, but it's just a bit too smooth. Season 5 is the pinnacle where they have just figured out the magic formula (which they didn't quite have down in Season 4 so it has some real rough misses), but they haven't gotten so practiced in it that it feels old hat (which Season 6 starts to smell of). The best episode ever is Season 5 - "Homer loves Flanders" and the second best is "Homer and Apu".

Also, not having watched The Simpsons is kind of like not ever having seen a Shakespeare play or heard The Beatles music. It's *the* seminal work of the medium of Televison.

11-19-09 | Myanmar

I'm going to SE Asia for Christmas, and I'm vaguely considering trying to go to Myanmar. (probably won't)

There's some who suggest that you shouldn't go because of the oppressive government, that tourist dollars help prop up their regime. I think this is basically nonsense. It's not a huge amount of money, and not much of that will get to the government; if you're a conscientious tourist you can try to use local currency and spend it at small local operations rather than the big tourist hotels and guided tours that kick back a lot to the government. I also think there's a ton of benefit to the local people and world awareness just to have contact with westerners (I would say the same about Cuba, etc. etc.) - isolating the people of the country is a terrible way to affect change ; I think we see that throughout history the best way to democratize countries is through maximum contact with western society.

If you actually want to do something about the Myanmar government, you should boycott Chevron, who pumps a ton of money directly to the regime for the natural gas leases in the north. We theoretically have economic sanctions against Myanmar, but of course the oil companies are exempt from that, just as they were when we had economic sanctions against Iraq. The result is that the people are impoverished but the government still gets plenty of money for its military, since the oil money goes directly to the government, while any smaller businesses that would help the people are not allowed to trade.

See : Amy Goodman on Chevron in Myanmar , Amnesty International calls for protesting Chevron and Chevron's response .

I heard the CEO of Chevron on NPR once responding to questions about the Myanmar lease. His justification was basically "if we don't do it, someone else will" (and the stuff in the above link about local schools, health care, blah blah blah). He's definitely right that someone else will. There's no oil/gas in the world that some oil company won't touch. Even if our government had balls and enforced real sanctions so that Chevron and Exxon were forbidden from taking the lease, surely China would step in. CNPC is already building a pipeline through Myanmar and is the biggest supporter of the regime already; if Chevron stepped out, CNPC would step in, and without the benevolent local spending.

Despite that reality, it's a preposterous moral argument that it's okay for us to be bad because someone else would be worse. In that situation you should clearly refrain from the bad action and also try to get the world to stop the worse action. We clearly need more international pressure on China to stop dealing with oppressive regimes around the world, but we also need to start doing so ourselves.

I am the worst kind of liberal (*). I can get all outraged about some indignity and rant about how people should behave, but when it comes down to what I personally want to do, I can always rationalize why it's okay for me to do it.

(* = the most common kind / the only kind )

11-18-09 | Information

Intellicast has awesome interactive weather maps. The best. Also, the UW ProbCast provides sophisticated computer modelled weather predictions for Seattle. The ranges show you the 90% confidence interval; mouse over things for more info.

Hot Map is just a bunch of scanned in paper maps that you can navigate around. SO SO superior to computer generated maps like Google. See for example Bangkok ; give me a good paper map and I'll beat anybody relying on electronic maps and GPS in a scavenger hunt.

It's funny reading about Julia Child and Craig Claiborne and that era of food in America, when you could become famous just by introducing America to some exotic new dish from foreign lands, like "spaghetti". It used to be so easy to travel and see things that you've never encountered before. It's such a fundamental part of human nature to explore and discover and bring things back, and it's basically entirely gone from our lives, which is a damn shame. These days if you discover something it's just because you are uneducated - you could have easily known about it before your journey if you were a better googler.

11-18-09 | Raw Conversion

CR2 format is a big mess of complication. My god I hate TIFF. The main thing is the sensor data. It appears to be stored as "lossless JPEG" which is a new format that uses the JPEG-LS predictor but then just codes the residual with normal JPEG Huffman coding. The sensor data is RGGB which they either store as a 4-channel per pixel [RGGB per pixel] or as 2-channel [GR or GB]. Either way is clearly not optimal. One interesting thing I could do if I cracked the CR2 format is store all these raws smaller with a better compressor. The RAWs from the S90 are around 11M on average, it uses the 2-channel mode; the RAWs are 1872x2784 = 3744x2784 samples and 12 bits per sample. That means the JPEG is getting to 8.85 bits per sample. Not very good.

Of course I probably have to use dcraw to read it for me, but dcraw is just about the worst piece of code I've ever seen in my life. It's a miracle to me that people are able to write functioning software from code like that.

Paul Lee has a modified dcraw and some nice sample pictures of how demosaicing can go wrong (click the Moire or Aliasing links).

My idea for high quality RAW processing :

First of all, abandon your idea of an "image" as a matrix (grid) of colors (aka a bitmap).

The S90 sensor has barrel distortion that's corrected in software.

It also samples colors in an RGGB Bayer mosaic pattern (like most cameras).

The two of those things combined mean that you really just have a collection of independent R's, G's, and B's at
irregular positions (not on a grid due to barrel distortion).

Now, you should also know that you need to do things like denoising on these original samples, NOT on
the grid of colors after conversion to a bitmap.

So I want to denoise directly on the source data of irregular color samples.
Denoising R & B should make use of the higher quality G data.

Denoising should of course use edge detection and other models of the image prior to make a Bayesian
maximum likelihood estimate of the sample without noise.

To output a bitmap you need to sample from this irregular lattice of samples (mosaic'ed and distorted).

Resampling creates aliasing and loss of information, so you only want to do it once ever on an image.

There's absolutely no a-priori reason why we should be resampling to the same resolution as the sensor
here.  You should resample at this point directly to the final resolution that you want your image.

For example with the S90 rather than outputting the stupid resolution 3648x2736, I would just output 3200x2400
which would let me view images at 1600x1200 on monitors with a box down-filter which will make them appear
much higher quality in practice (vs 3648x2736 viewed at 1600x1200 which involves a nasty blurring down-filter).

The output from this should be a floating point bitmap so that we don't throw away any color resolution

Exposure correction can then be done on the floating point bitmap without worrying about the irregular
lattice or any further resampling issues.

11-17-09 | Cleaning

I did a full clean up of my laptop and it's running like a million times better now. There are lots of badly behaved apps on Windows that seem to just crud up your disk with temp files so badly that they make themselves slow. Do this on a fragmented disk and it's slowness death.

How to fix :

1. Uninstall badly behaving apps (* see later)
2. Del temp files and caches (* see later)
3. Defrag disk
4. Reinstall bad apps

The key thing is getting your disk free of all the shit before the defrag, so that the shit files aren't getting mixed in with the real stuff at the front of the disk when you defrag.

Here's my temp file cleaner :

echo dir c:\temp :
call d -r  c:\temp\*
REM call d -r "%USERPROFILE%\local settings\temp\*"
REM call d -r "%USERPROFILE%\local settings\temporary internet files\*"
call zdel -y -r  c:\temp\*
call zdel -y -r "%USERPROFILE%\local settings\temp\*"
call zdel -y -r "%USERPROFILE%\local settings\temporary internet files\*"

REM call zdel -y -r "C:\Program Files\Visual Assist.NET\vc7\history\*"
REM call zdel -y -r "C:\Program Files\Visual Assist.NET\vc7\cache\*"

REM call zdel -y -r  "c:\windows\system32\config\systemprofile\local settings\temp"
REM call zdel -y -r  "%USERPROFILE%\application data\acd systems\acdsee\imagedb.*"
REM call zdel -y -r  "%USERPROFILE%\Local Settings\Application Data\VisualAssist"

(d is a dir and zdel is a delete ; -r is recursive).

Visual Assist and ACDSee are two apps I use regularly which are fucking retarded in that they seem to just grow their caches on disk indefinately until they get so big that it badly affects their startup time. My ACDSee was taking a few seconds to start up and I was like "WTF" and found the ImageDB cache was a few hundred MB; delete that mofo and it starts up fast again. You probably don't want to do those REM'ed out lines every time you clean up junk (it will make VA do a full reparse), but they're good to do once a month or so.

(*) badly behaving apps :

There are a few apps that seem to crud up your disk super badly with temp files; you may even have trouble finding them all; they're mostly under "Documents and Settings" , spread around in Local Settings and Application Data. With all of the following apps you should first uninstall them, then go through these temp dirs and find as many of the files they left behind as you can and delete them, becuase they don't clean up after themselves correctly. The worst offenders that I've seen are :

1. Firefox. So impossible to make this beast clean itself. If your Firefox is taking a long time to start up, best move is to uninstall it. It will leave lots of shit behind, delete that manually.

2. Adobe Everything. I had tons of shit from Acrobat Reader 6, Acrobat 7, and Acrobat 8, even though I uninstalled all of them (Foxit FTW). Uninstall all and delete manually.

3. Apple Everything. Uninstall all Apple software, delete all the shit they leave behind, then do yourself a favor and don't reinstall it. There was almost a gig of mystery shit in Apple folders, including shit like copies of installation packages of various versions of iTunes. No I don't want you to backup the install packages for me thank you very much.

In all I freed up about 10 Gig from this, and getting rid of all the shit before the defrag lets the defrag do better work.

ps : may as well run Malwarebytes while you're at it.

pps : may as well delete the masses of VC shit too :

zdel -r -y "*.(ncb|ilk|idb|opt|plg|dsw|pch)"

ppps : may as well checkpoint your p4 server too :

p4d -jc -z

11-16-09 | Warning 4244

I wish I could disable warnings about float - double conversions but keep warnings about the conversions between different types of ints. I hate that every time I call sqrt() or whatever in some utility code I have to spend a bunch of time worrying about whether I'm in float or double or whatever, and every time I write FPU utility code I have to write versions for float & double, or cast at the call site, bleck blah lame.

For int - int type conversions I'm trying to be good these days and use stuff like these :

// check_value_cast just does a static_cast and makes sure you didn't wreck the value
template < typename t_to, typename t_fm>
t_to check_value_cast( const t_fm & from )
    t_to to = static_cast< t_to >(from);
    RR_ASSERT( static_cast< t_fm >(to) == from );
    return to;

#define RR_CLAMP_U8(x)  (U8)RR_CLAMP(x,0,0xFF)
#define RR_CLAMP_U16(x) (U16)RR_CLAMP(x,0,0xFFFF)

So eg.

S32 val = whatever;

U8 t1 = check_value_cast< U8 >(val); // no clamp but asserts that it fits in range

U8 t2 = RR_CLAMP_U8(val); // clamps to target type range

Losing values to type conversion is one of the few nasty bugs I continue to have, and I'd like it to stop.

11-15-09 | New York Times Magazine

The New York Times Magazine this week has an article on video games as "art" (See the article here ). It mainly focuses on Jason Rohrer and his game "Passage" (see it here ).

I came here to write an angry rant about how retarded the article is and how Passage is absolutely the worst kind of attempt at "art" and the worst kind of "game" (in that it's not a game at all - it's a "toy" or an "experience" ; at the Indie Game Jam we always tried to have a rule that we didn't just make "toys" or "experiences" because they're easy and uninteresting).

But then I realized that actually the article and Passage are both actually brilliant on a meta level. The article is by someone who totally doesn't understand games or art, but is trying to seem intelligent and writes a lot of pretentious nonsense. Rohrer is either an idiot or is intentionally spouting bullshit to make Passage and games seem more significant than they really are. The article is aimed at out-of-touch old people who will nod and go "hmm interesting" and then share what they read at some cocktail party and everyone will pretend to agree in order to seem smart. That's art! It's exactly like the traditional art world ! How brilliant, by being retards they have shown that games are art.

(I realized this while walking around the SAM looking at the Calder mobiles and being amazed by how "artists" can create this amazing mass delusion to make critics talk about all the powerful themes in their work that don't really exist).

This article was part of yet another weekly new low for the NYTM. This week the main articles are on Megan Fox and Octomom. It's literally a tabloid now, but even worse than a tabloid because the gossip and tawdry condescenion is cloaked in a language of detached observation and pretentiousness. It's like "oh we're way too smart to actually care about Megan Fox and Octomom so we will write about the phenomenon of other people being interested in them".

Of course that's all people really want. People don't want to actually be intellectually challenged. They just want confirmation of their existing beliefs. They want opinion without facts. They want sex and gossip and soap operas. If they're sophisticated, they like to get their trash cloaked in a veil of higher culture (see, eg. Mad Men, 2666, etc.).

.. also Passage does indeed invoke powerful emotions. It doesn't make you think about mortality or futility unless you're a fucking moron. But it does make you feel frustration, disgust, disbelief, anger, and finally sadness that you wasted your time on this pile of turd and that our cultural critics fall for this nonsense so easily.

11-15-09 | Braised Pork

Night 1 : Braised poker shoulder with polenta. About 4 hours at 325. I added about a cup of root beer this time which had a nice flavor but too much sugar. White wine is really the best braise/deglaze liquid IMO. Also I think it may be best to chop up the meat before browning because it gives you more surface area to get delicious brown on.

Night 2 : leftovers. On night 1 when you put it in the fridge, pour off all the liquid into a tall narrow glass so that the juices and fat separate neatly. Take the fat off and reserve. The juices should be totally solid and jello-like when cold, indicating you have rendered out the collagen. To make a sauce, take just the quantity of juices you want for the moment, reduce the juices a bit in a pan, then whisk in butter. This is a general sauce boosting technique which will make any homey braise liquid more luxurious.

Night 3 : meat pies (Pastys, Empanadas, whatever you want to call them). I've tried these a few times and they haven't been awesome, but I think I finally nailed it. There are lots of little details to get right :

1. Preheat oven to 375 with a baking stone in it. You should prep the pies on parchment paper and then just slide that onto the stone. This will cook the bottom quickly on the preheated stone which keeps the bottoms of the pies from getting soggy.

2. Forget the dough you are "supposed" to use for meat pies (be it the dull Welsh Pasty dough or the puff pasty dough or whatever, they all suck). Just make the crust that tastes delicious to you. I like Martha Stewart's Pate Brisee (BTW any time you see a hat in French it means there's an "s" there in the English version, so pate = paste, hopital = hospital, etc. ) . To make it savory instead of sweet : A. omit the sugar ; B. double the salt (or just use salted butter + the called for salt) ; C. add some pepper and herbs to the crust ; D. use a bit of the separated chilled fat from the braise for some of the butter. (note : I generally think frozen pie crust is an okay short cut for pies, but they are all sweetened so cannot be used here; plus the whole point is the delicious crust, so you're really only cheating yourself that way). (of course do all the things you are supposed to for pie crust - work very cold, do it all in advance so they have time to chill before rolling out, etc.)

3. For the filling : the main thing is it should all be pre-cooked and not have very much moisture. I tried a few things, the best filling I did was like this : cut the pork shoulder into bite size chunks; saute some onions and carrot until just al dente (it will cook a bit more inside the pie but not much) ; let it all cool and mix together. Take a few tablespoons of the mixture and pulse in food pro to make it into a paste, crack in one whole egg, stir this back into the chunky mix as binder. This should look like a very chunky meat loaf or meat ball preparation. Spoon into pie. Be very liberal with spicing - the crust will mellow out the flavors so the filling should be full of punch.

4. Roll out pie crusts quite thin - thinner than you think is okay. Don't over-fill, leave a good inch of border for crimping. Paint edges with water to seal, then crimp well. Cut three BIG air vents - they need to be big because the crust will puff a bit and try to seal itself back up. Brush with egg wash and sprinkle on a bit of coarse sea salt. Bake about 30 or until colored. Brush with melted fat near the end of baking if you want a bit of glaze.

They look like this ; BTW I'm pretty pleased with the S90. It's got its flaws, but it takes pretty amazing photos when you just put it in Auto and press the button and don't do anything (like this one) which is what you really want from a little P&S.

Night 4 : Packaged pork Ramen with pork chunks and cloves added for lunch. Tacos of pork chunks for dinner. (the key thing here is to cut the pork into chunks and pan fry it in hot oil to make it really crispy, like how carnitas is done). Om nom nom.

Night 5 : fear food poisoning risk and throw out last bit of pork.

11-14-09 | Stereotypes

I cop out of fashion choices. I like to just wear jeans and t-shirts because it shows that I'm not trying. When girls ask me about fashion things I say "I dunno, I suck at fashion, don't ask me". Of course the real issue is that I'm afraid if I try a little bit, it will just look shitty and look like I'm trying and failing. I don't want to put in the time and effort to really do a good job, so instead I just intentionally do a really bad job.

This is a shitty position to be in (not wanting to try all the way so you intentionally don't try at all). It would be much better if you could just try a little bit. The problem is that it works so well. When you try a little bit, the world mocks you. They don't appreciate the fact that you're making an effort, showing what you can do, even if you suck.

I think when girls do the whole "do I look fat?" or "you don't really love me" routines, obviously it's partly insecurity, but it's also a dominance game. They get you to prostate yourself before them, to whimper and beg them to feel okay, they humiliate you, and then if you have performed like the lowly dog that you are to their liking, they show mercy and forgive you.

Women use their emotional powers to manipulate and attack men; when women feel vulnerable or demeaned they lash out with painful words. This is a balance for the physical dominance of the male; we could beat them up at any time and they're a little terrified of that and try to defend themselves and achieve their own sort of power in a relationship. It's only fair, I don't think I'm saying anything controversial; men are very naive about interpersonal war, most women are far more adept at it, and of course they use their skills for their benefit. Women use their sly emotional dominance skills on other women of course, but also on their lover, something that most men find terrifying and revolting. There's a power gap - both sides know the man is physically more powerful, and that's a shitty feeling for the woman to have, even if you never use that power. It's sort of like a non-nuclear state being in negotations with a nuclear state. Like hey, we know you're never gonna use the nukes, but dude you have a picture of one on the wall, that makes us feel a bit uncomfortable here.

Digital ovens are such fucking retarded bollocks. I shouldn't have to hit Plus/Minus to set the fucking temperature. In order to put my oven to Bake at 400 I have to hit something like 10 key presses; I have to hit "bake" "temp set" "plus plus plus plus plus" "bake". It should be a fucking spinny knob like the old days. And my fucking OVEN SHOULD NOT BEEP AT ME EVER NEVER EVER !! Not when I press buttons, not when it reaches the desired temperature, never ever never. Fucking electronics everything flashing and blinking and beeping is a fucking awful noise polluting annoying turd. Turn off all the lights in a house these days and it's fucking red and green christmas light city of blinks everywhere. The old plain mechanical ovens with an analog dial you turn to set temperature were just about perfect in terms of user interface and usability. You modern designers all suck so damn bad.

11-13-09 | Noise

I really hate the trend for heavy noise reduction in photos these days. Heavy noise reduction gives everything a weird playboy-airbrushed kind of look with big flat solid color patches and super crisp edges. It's fucking gross. It's tacky, it's tasteless. It's like the fucking photos that people make black and white except for one object left in color. (It's almost as bad as all the mangled HDR that people are doing to make these gross artificial-color shots). Makes me want to fucking vomit.

Canon have played a dirty trick with the S90 by putting some pretty heavy noise reduction into the firmware that becomes stronger at higher ISO. Of course that's sort of what you want, but you want to do it outside of the camera ; noise reduction is a lossy operation that kills information.

The worse thing is that all the review sites play right into their hands. They will mention the S90's strong noise reduction, but they also show direct side-by-side images from cameras , eg : S90 vs LX3 and then say "the Canon shows significantly less noise at high ISO". Well fucking DUH. You aren't comparing apples to apples. Then you have geniuses like this guy who say they are doing a fair test because they don't run outside noise reduction software. Well, no that's not a fair test if one of the cameras has more noise reduction firmware. It is clear that the S90 has some kind of newer fancy noise reduction alg in the firmware, like bilateral filtering or something. There's no such thing as comparing the "raw image that the camera makes" any more by not doing post-processing in software, because the cameras all cheat and do different tricks in firmware to make their images look better.

Anyhoo. I was pondering why digital camera noise looks so objectionable. Real film noise doesn't look awful. If you take shots without enough light on a film camera, you get a nice speckly grain. Digital cameras look much worse. I think a big part of it is the uneven color matrix thing that DC's use which gives the noise ugly chroma shift speckles. The only other things I can think of are the clamping to 255 and the fact that we JPEG after noise which smears it into DCT basis functions.

One thing that's a bit perplexing is that printing noisey digicam photos makes them look a million times better. Again I'm not sure exactly why all this is. Surely one reason is that printed pixels are actually little ink blobs that run into each other and blur; this adds an extra physical level of some kind of gaussian like blur, but it's better than any digital blur you can do because it's operating at near-infinite subpixel resolution (aka the real world).

The S90 has a low-light mode that makes images with 1/4 as many pixels, presumably because they're binning 4 buckets on the sensor together to get more light per pel. I wonder if they're actually doing anything better there than what could be done as a post-process. BTW it occurs to me that you could make a much higher quality black & white digital camera if if actually had a sensor with no color filters on it, so every bucket was open to any type of photon; it would have a true pixel at every bucket instead of the funny chroma selection that color digicams have.

11-13-09 | The Pixies

The Pixies are touring now. I think when you go to see a band relive their glory days even though they sound awful now, that's the defining moment when you become a pathetic old fogie.

11-11-09 | Google Voice etc

I have found one great use for Google Voice : I can give that number as my business contact number (for banks, credit cards, etc.) and then set it so that those people go straight to voice mail and never rings through.

Luminaplex algorithm for improved generation of luma subsampling. I guess it's the obvious answer that we will use for everything in the present/future now that computers are fast. Rather than try to specify a forward and inverse transform, just specify the inverse transform and then do a brute force optimization to find the data which reproduces the original as well as possible after inverse transform.

However, Luminaplex is a bit silly. If you're serious about fixing chroma nastiness, the right fix is to use the Luma edge information in the chroma-upsample. I'm not aware of anyone doing this in the main stream, but obviously high quality JPEG decoders should be doing this. In fact there are lots of things you could do to make a high quality JPEG decoder which would actually be a pretty useful thing since we're stuck with this evil format.

In other news : Land's End has gone to shit, don't buy anything there any more. I used to like it because it's one of the few places you can get "tall" sizes, so my freakish monkey arms wouldn't hang out six inches past the end of the sleeves, but suddenly they seem to have switched their production to be super cheapo Chinese shit and they've partnered with Kmart or Sears or some shit. Garbage.

I also just got my Canon S90 today. It was either that or a Panasonic LX3, I couldn't make up my mind so I flipped a mental coin. First impression : URG , I really don't like the ergonomics of it. I hate not having a view finder, but the LX3 doesn't have one either so I was fucked on that. The shitty thing is it's so small and it's got buttons everywhere on it that I don't feel like there's anything to hold it by; I feel like I'm holding a faberge egg, I can't put my fingers anywhere without touching a button by mistake. It's also smooth all over, it has no nice nubby grippy bit; I kind of want to superglue on a strip of rubber. Yeah the control ring is obviously sweet, but I would gladly throw that away for a better grip surface like the LX3 has. Will report again later with more impressions once I take some real shots.

It's still nowhere near my dream digital camera. Some basic attributes of my dream camera :

Tiny. The LX3 is about the maximum size (the S90 is a bit smaller than the LX3, definitely an okay size).

Minimal in-camera processing. Shoot RAW and let me do noise reduction / white balance / etc. in software after the fact. I want the camera firmware to be as simple and direct as possible with as few modes as possible exposed. Basically the only fancy thing I want done in camera is stabilization. This removes the need for a bunch of settings in the camera menus and lets the software get better over time.

Fast. In manual focus mode it should be super super fast, like 0.1 seconds or less cycle time between shots. There's absolutely no reason for all the delays in cameras these days. It should never stall out because of buffer flushing either; WTF is that, memory is fast and cheap.

Good optical viewfinder. Ability to turn the LCD off completely all the time would be nice. Large LCD and image review is completely irrelevant to me; I just want to snap the photos, I'll review them on my computer. In fact I'd be content with no LCD and no review at all if it meant a smaller lighter camera with longer battery life.

Big sensor and lens that lets in lots of light, not too many pixels. I want good low light performance. The Canon S90 is supposed to be okay on this, but I'd like even more. I'd be happy if the camera had no flash at all, it saves me from having to turn it off, saves a mode to fuck with, and reduces size and weight. Has there ever been a flash photo in the history of the universe that was worth keeping? I don't think so. (not in camera flash anyway, external flashes can be okay).

BTW the Leica D-LUX 4 is a pretty hilarious product. It's one of those clear "I'm an absolute moron" purchases. It's literally identical to the Panasonic LX3 (except that it's *worse* because the front is smooth while the LX3 has a nice old fashioned grip style) - and everybody knows it's identical to the LX3, and they charge almost twice as much for it, and yet people buy it. Wow.

11-10-09 | Truth

Truth is the hobgoblin of small minds. Foolish people dismiss things because they aren't true.

A while ago we talked about the myth of the Bonobo. The myth is that humans are most closely related to Bonobos and Chimps. Chimps are super violent, aggressive, territorial, masculine ; they settle disputes by beating each other; when conflicting tribes of chimps meet at a border they scream and often kill each other. Bonobos are peaceful, tender, feminine; they settle disputes by fucking; when tribes meet they groom each other and fuck. It's almost like the two sides of human nature split in two (like the Dark Crystal) to form the Bonobos and Chimps. Of course it's a big lie spread by sloppy biologists who just saw what they wanted to see, but who cares? It's a fucking great story. I despise it when I tell that story and some jackass is like "that's not actually true". I know it's not fucking true, pipe down.

I have a great fondness for romanticized peasant farmers, like in Tree of Wooden Clogs or Once in Europa or whatever. People just love to point out to me that it's not really like that; that those people are actually just provincial and ignorant and awful rednecks. I fucking know that! I'm not retarded, I know it's romanticized ; is any movie or book or anything remotely realistic !?

Another one that bugs me is when somebody's trying to prove a point and they cite some examples or some history and they fuck up the specifics, some jackass will always point out the little mistake they made. You're wasting our time and detracting from the real argument. Just pretend that he cited the right examples and then focus on the point that he was trying to make.

11-10-09 | Touchy

It's always perplexed me how touchy people are about things that don't matter. Like yeah, I understand when you're touchy about something you care about or are sensitive about. Like if you hope to be a writer and you give me something to read, okay, I know you're touchy about that and that's fine. (of course in that situation I feel just crippled, afraid to say anything, it's so awkward, I hate it when people ask for my opinion on their work because I know they don't really want it).

People who absolutely suck at cooking are totally touchy about it. If you just ask simple questions like "is this Lawry's seasoned salt on here?" they get all defensive like "my momma used Lawry's , there's nothing fucking wrong with Lawry's" ; well A. chill out, I didn't say there was anything wrong with it, B. yes there is something wrong with it, and C. you obviously don't fucking care about cooking so why are you so defensive? D. it was actually pretty fucking delicious and I was going to compliment it before you lost your shit.

A funny one that seems really universal is loading the dishwasher. God forbid you rearrange someone's dishwasher loading because they did it badly, they'll be all like "oh, my dishwasher loading's not good enough for you? uh you're so annoying and picky". Well if you fucking had any clue about how dishwashers worked you wouldn't put pans right in the way of the telescoping jet that comes up in the middle, and you wouldn't put the plate surfaces right up against each other so that no water can get between them, and you wouldn't put bowls on the plate rack, or lie wine glasses down sideways. I mean

11-10-09 | Sustainable

"Sustainable" is the new fucking ridiculous tag-word that's used to sell shit.

First of all, anything super expensive is not sustainable. That's almost a tautology - if the majority of the people can't ever afford it, it does nothing for large scale effects of humanity on the environment, nor does it improve most people's lives. Fucking hand-fed berkshire pork is not "sustainable" - it's a fucking indulgence. Small farmers in general are a massive drain on the environment; large scale farming is of course the only way to be sustainable; see aso :

People use "sustainable" a lot to mean "hand made" or "old fashioned" ; like "sustainable wood carved toys instead of mass produced junk from China". Hey, I love hand carved wood children's toys, but "sustainable" ? Hell no. First of all, anything made from wood is not sustainable (a common misuse of the word). But most of all, anything involving a lot of human labor is not sustainable, because it makes it expensive, and it actually makes it massively polluting and gives it a huge carbon cost. A human life has a huge carbon cost (and general huge environmental impact). Anything hand made is massively damaging to the world, and thus again in no way sustainable.

The right way for us to make a sustainable existance is to of course take advantage of the efficiency multipliers of factories and mass production. Anything that produces less product and takes more time and labor is almost inherently less sustainable; there have to be massive other confounding factors for it to make up for it.

Of course if you want "sustainable" toys, the real place to get them is at a thrift store. Reusing is sustainable, cutting off the cycle of constant production of new crap that just depletes resources and increases and toxins and just gets thrown out. But of course the "sustainability" yuppies are just fad-followers so they will buy all new junk and then throw it out when the next fad comes along.

11-09-09 | Health Care 2

Some data :

nice scatterplot that shows our outlier clearly

tons of nice charts here but you have to click on them to read them.

More data here ; in particular Exhibit 6 at the bottom is the key problem. Health care spending as a percentage of GDP is growing fast in Europe. (not as fast as the US, but still bad).

Another key point I think is that the US system is horribly broken at the low level. We often focus on the problem of who's paying (insurers vs. government) and who's covered (univeral, poor, etc). But even once people get into a hospital our system is fucked. Medicare + Medicaid (+ all government spending on health care) is 7% of GDP, which is close to what the Europeans spend on all their health care. Government spending is also growing as a percent of GDP (even normalized to remove the growth of the elderly population), so it's not like that is controlling costs. I think it's easy for politicians to villify the private insurers, but they are only a small part of the problem. We need to be looking at doctors who are making poor care decisions.

ADDENDUM : This Uwe Reinhardt guy is alright. He annoyingly uses the hack's method of presenting only little snippets of data that are carefully chosen to prove his point at the time, but it's still interesting.

Uwe E. Reinhardt - Economix Blog - NYTimes.com

Why Does U.S. Health Care Cost So Much (Part I) - Economix Blog - NYTimes.com
Why Does U.S. Health Care Cost So Much (Part II Indefensible Administrative Costs) - Economix Blog - NYTimes.com
Why Does U.S. Health Care Cost So Much (Part III An Aging Population Isn’t the Reason) - Economix Blog - NYTimes.com
Why Does U.S. Health Care Cost So Much (Part IV A Primer on Medicare) - Economix Blog - NYTimes.com
U.S. Health Care Costs, Part V Can Americans Afford Medicare - Economix Blog - NYTimes.com

Who Needs the Public Option - Economix Blog - NYTimes.com
Is Medicare Raising Prices for the Privately Insured - Economix Blog - NYTimes.com
Health Reform Without a Public Plan The German Model - Economix Blog - NYTimes.com
A ‘Common Sense’ American Health Reform Plan - Economix Blog - NYTimes.com

11-08-09 | Health Care

Well health care reform is moving forward, and it's a huge pointless disaster as predicted. We're going to basically wind up covering more people within the same current broken system. I actually think things like not letting health insurers refuse to cover certain people is a very negative development, as is mandatory coverage and forcing employers to cover people. (I've written before about why I think the whole connection of employers and health care is very negative for our economy and personal liberty in general).

I was thinking about what we really should do for health care, and it occurred to me that it's really even a larger problem. Health care is definitely the biggest single manifestation of this issue, but at the bottom the problem is manipulative binding contracts, exclusionary pricing that forces you into contracts, closed networks, and collusive oligopolies. These same issues are afflicting people in bank accounts, cell phone contracts, cable networks, etc.

The real solution is to find a way to force all of these things to open up to consumer choice. The big problem with health care really is : consumers have no realistic choice options - all health insurances are basically the same ; consumers have no access to information to choose doctors based on prior performance or based on cost ; doctors have no ability to charge different amounts and are not rewarded for charging less or doing less ; specialists are rewarded heavily for doing unnecessary work ; consumers can never make a cost/benefit choice to opt out of care for the elderly or very ill ; consumers can't change plans or opt out of plans because of the collusionary pricing that forces you to have health insurance and not pay cash ; etc. etc.

Basically there is no capitalist competitive market at work in health care at all. Doctors/hospitals are not motivated to do better cheaper more efficient service to attract more clients. Health insurance is not motivated to charge rates that are competitive with cash rack rates because that option does not exist. Health insurance is not really "insurance" at all for most people, but rather just a huge fee that you have to pay simply to be allowed to get care.

With all of these things - bank accounts, cell phones, credit cards, health care, cable networks - I'd like to see a few major overhauls. 1) legal requirement to provide reasonable cash "rack rates". That is, consumer should be able to choose not to sign a binding contract for services, and instead just pay-per-use at a reasonable rate. 2) networks should be required to be opened to other companies at reasonable rates; eg. credit card point of sale networks, ATM's, cable wires, health care providers, etc. should all be required to allow third party companies access. 3) requirement to provide quick clear reports of past performance and disclosure of billing and furthermore notificiation of billing. For example, cell phones and banks should have to disclose how much the average consumer actually pays for bills; eg. when they advertise "$29 a month" cell phone contract, but without including all kinds of surcharges and fees and extra charges, they should be required to disclose "average consumer actually pays $63 a month". Also, when an unexpected bill arises they should be required to notify you *immediately* and even get permission, not just send you a bill months later. eg. if you make some roaming call they should have to say "this will cost $1 do you want to proceed?" not just send you a bill for $1000 at the end of the month. Similarly banks with overdraft, health care with out-of-network anaesthesiologists, etc. There's absolutely no reason not to provide warning these days with computerized everything and automatic email & SMS warnings.

You can't have working capitalism without informed consumers, and you can't have working capitalism without consumers being free to change services. Currently all of these industries are completely broken and are doing nothing but intentionally suckering people into contracts where they pay outrageous fees with very little ability to avoid it or get better treatment.

Really I'd like to avoid a lot of federal regulation, what I'd love to see is for it just to open up to competition, but in some cases the best option may be regulation. For example with cell networks, I think a good argument could be made for just nationalizing the networks and making them open at auction rates to 3rd party service providers. Another option would be to make it illegal to tie a phone to a given network, which would allow consumers to change provider at any time, which would make network bandwidth and quality a commodity and greatly improve the market.

Many of these services have also basically become "utilities" that a human must have to function decently in this society. You can't really choose not to have a bank account or health insurance or a cell phone, and that argues for the government providing them, or mandating some basic availability by law. For example cell providers could be required to provide a $10/month text & voice only contract for the poor.

Banks in particular make me angry because even without the recent TARP disaster, they get massive amounts of public support through the free money from the Fed and the insurance from the FDIC. I think it's perfectly reasonable to have very high requirements on any bank that takes Fed money. For example, they should be required to provide a very basic checking account with zero fees, and most importantly no line or credit or overdraft account, which are extremely predatory practices forced on the poor (and no NSF fees either up to 3 bounces or something; and of course automated warnings).

But of course we'll never see any of that, because the government *wants* corporations to rape consumers, and doesn't actually want to see better competition that will improve services.

11-06-09 | The Basics

People often ask me how to lose weight. I ask them "have you done a log of your diet? writing down everything you eat for a few days?" and "are you writing down your workouts and recording the intensity?" . The answer is invariably no.

It's just the obvious basic fundamental thing to do. People always want some magical new solution, some amazing wisdom they didn't think of that will solve all their problems. They are also always fixated on the hip new solution that just came out this season, when in fact the obvious old fashioned way is almost always just the right thing to do.

People will ask me how to improve their data compressor, or speed up their slow code. Well, have you measured it against some other known standard ways of doing things? Have you set it up to be repeatable and take standard data sets as input so you can make reliable tests? Have you put in counters to analyze the parts to know where you might be going wrong? If not, I don't even want to talk to you about it. Come back when you've done the obvious basics.

Broke people are always struggling with their finances and can't figure out how to get by on their low salaries. Well, have you done a budget? Are you writing down everything that you spend money on? Do you have a log of all the necessary expenses that are upcoming? The answer is invariably no. Well, that might be a good place to start, you know?

11-06-09 | Dance

Modern Dance is only tolerable when there are hot mostly-naked chicks in it. It's not that I'm interested just for the naked chicks, if I wanted naked chicks I'd just look at naked chicks - I'm in it for the dance, the movement, the forms, the light and shadow, the interaction with the sound. It's just that all those things are not enough; if you have those things without hot semi-naked chicks, it's not enough value to get me out of the house. The hot naked chick is a the bonus added value that puts it over the line. You could say the same thing of relationships I guess.

11-06-09 | IsSameFile

I found myself wanting to know if two file names were the same file on disk. It's hard to check that just by looking at the name. Obviously you have issues like one might be absolute, one might be relative. Even if you fix that, they could be different A-code-pageizations of unicode names. And something I hit often is one of them might be on a "subst" or even a hard link. I want to know if they are actually the same file.

This appears to work :

bool IsSameFile(char * Name1,char * Name2)
    HANDLE f1,f2;
    if ( f1 == INVALID_HANDLE_VALUE )
        return false;
    if ( f2 == INVALID_HANDLE_VALUE )
        return false;
    // BY_HANDLE_FILE_INFORMATION has a unique file ID and Volume Serial Number in it
    //  check those are the same
    //  heh fuck it just check they are all the same
    // confirmed : this does work across substs
    return memcmp(&info1,&info2,sizeof(info1)) == 0;

11-06-09 | I-Phone

Over the weekend in SF I finally used an iPhone myself extensively to look up clubs, restuarants, etc.

OMG what a piece of shit. All you iPhone users have just been lying to me about how great it is; are you all brain-washed !? If it belonged to me I would have bashed it against a desk or thrown it out a window it was so fucking frustrating to use.

1. The fucking finger navigation is so broken. I kept accidentally clicking links. It won't fucking let you scroll around when the page is still loading !? WTF !? How do you put your finger down to navigate when the spot you need to put it on is all covered in links? Plus the amount that you can scroll is only equal to one page.

2. The two-finger zoom is a fucking useless gimmick. So annoying and hard to use. Again I'm accidentally clicking links or scrolling, and I constantly run into the limit of how much my fingers can move on the screen. It would be so much better just to have a mouse-wheel type of zoom wheel, or even a +/- magnifying glass increment. Hell for scrolling I'd rather have two physical sliders or wheels on the edges of the screen.

3. The keyboard thing is mostly okay, but occasionally very frustrating when you keep getting the wrong letter and have to back up over and over. Maybe it should zoom up the area that you are trying to click, like the tap should be a hold & release, it should select the letter on release and it should zoom while you hold, then you could hold & slide your finger to disambiguate.

4. No Flash support !? WTF. I knew about this, but so much of the web is flash-only now, it's pretty crippling.

5. No right click "open in new window" for links, and no way to duplicate the current page to a new page !? Maybe you can do these and we just didn't know how.

6. I often found myself in situtations where I couldn't go back in browsing. I'm not sure exactly how this happened but I'd just be browsing a bit and realize that I want to go back and then I notice the back button is greyed out and I can't get back. WTF. Combined with trying to scroll around and accidentally clicking links this was probably the biggest frustration, I'd have to restart my whole browse from the beginning because I'd misclick a link and then find I couldn't undo it.

7. No undo stack in general. All the scrolling and zooming and everything should be on an undo stack. Again largely needed due to awful laggy interface, you find yourself in weird spots and just want to go back and you can't.

8. Apps were even worse. I tried the Yelp app and the Urbanspoon app. Both have kind of nice interfaces, but it's just a fucking retarded idea. You're locked into their way of navigating and it's frustrating as all hell and they're missing features. I tried to fight with them briefly before screaming and just closing them and going back to the normal web. I don't understand choosing to limit myself into some dumb fuck developer's way of getting information.

Conclusion : it's a frustrating pile of turd and I don't know WTF you all are so happy about. It gives me great rage.

11-05-09 | Quicksilver

"Quicksilver" (starring Kevin Bacon) is a "classic" bicycle movie, in the sense that there are only like 3 bicycle movies and it's one of them. It's mostly just a light piece of trash featuring some pretty amusing absurdly cheezy 80's-isms. The dance scene where Bacon dances with his bicycle is precious.

Winning is a feeling you never lose.

Speeding towards the final -- and deadly -- showdown...

Kevin Bacon is Jack Casey. He traded in his three-piece suit for a ten-speed and the streets.

For Jack Casey, playing the market was life in the fast lane... until he joined QUICKSILVER where the fast lane was a way of life!

It also has the most insane boldly obviously wrong stuff I've ever seen in a Hollywood movie. The city shots randomly switch between New York and SF. And not like in a subtle way at all, like they specifically show very unique SF hilly streets, and then very NY dense traffic city streets. People talk in NY accents and the characters are obviously very NY and the messengers talk about NY addresses for deliveries (68th St., etc), but he works on the Pacific Stock Exchange. It's bizarre, it's like they wanted certain aspects of both cities and someone in the production meeting was like "oh yeah, we'll just switch back and forth, nobody will notice, movie-goers are idiots!". There are a few scenes where they literally ride around a corner in NY and turn onto a street in SF (and the lighting changes dramatically of course).

Bacon's bike also magically switches between fixed gear and freewheel. In most of the close up and low speed shots it's fixed, but then any time they're going fast or going down hill he's suddenly magically free wheeling.

11-04-09 | Video is not for Windows

Holy crap video is a flaming mess.

First, if you don't know, there are two main things : packages & streams. The packages are AVI, MKV, MP4, MOV ; they put together image data and audio data in some way (the layer that unpacks these is often called a "splitter" or "demuxer"). The packages send their streams to codecs which convert them to some format for display. On Windows this is all supposed to go through DirectShow which is supposed to use the 4CC codes and some priority information to automatically find the right handler for the various streams and packages. In theory.

The first problem you hit on Windows is that AVI packages are handled pretty well, but AVI can't hold H264 video because AVI can't handle the flexible B-frame ordering that H264 can generate. (limitted profiles of H264 can be put in AVI, and there are hacks around this problem, but you're getting into a world of hurt). So you need MKV or MP4 boxes, and those are handled poorly; some apps handle them okay, some don't. Some apps "cheat" and don't trust DirectShow like they're supposed to (the cheating apps often work better).

Things I've installed lately :

MP4Box : MP4 stream boxer/unboxer ; pretty decent app, recommended, but help is poor

YAMB : GUI for MP4Box.  Useful to help figure out command lines for MP4Box because the help is bad.
    YAMB has bad bugs though and will fail to launch MP4Box, so you have to copy out the command line
    and run it yourself

MKVVerify : MKV stream checker.  Useful because MKV support is so fucking borked.

MediaInfo : Media info reporter.  Questionable usefulness because I don't trust it and don't know where
    it's getting it's info for.

Graphedit : DirectShow graph visualizer and tester from MS

GSpot : AVI info tool.  Useless.

MSU VMT : Moscow State University Video Quality Measurement Tool.  This is pretty neat when it works,
    but far too often it fails to get the frames correctly, so you get totally bogus results.

MSU LS Codec : Moscow State University Lossless Codec.  Best compressing lossless codec, seems nice
    but crashes some tools when you try to use videos compressed with this.  Thus useless.

Lagarith Codec : This appears to be the one good working lossless codec.  Recommended.

HuffYUV Codec : Videos made with this crash me on read.  Jeff says it works great for him.  Avoid.

MeGUI : GUI for "mencoder" which can driver AviSynth and x264 ; like all of these big GUIs that try to
    run a bunch of other products, this mysteriously fails for me.  It seems to set everything up right
    and then it launches ten other programs and they fail to hook up in the way MeGUI expected them to.

Handbrake : see MeGUI

FFDShow : hooks up the Linux video decoders (ffmpeg , libavc, etc.) to DirectShow.  This thing is
    pretty evil and fails to report frame rate and media info sometimes, but is also the only real

Haali Media Splitter : MKV unboxer, works with FFDShow.  Difficult to install correct manually.
    Even when installed correctly, does some weird shit with framerate; doesn't seem to report it
    correctly through DirectShow.  Probably best to get a codec pack like :

K-Lite Codec Pack : works for me but generally is considered malware
Matroska Codec Pack : didn't work for me
CCCP Codec Pack : not tried

MPlayer : Linux media player, now ported to Windows ; very flexible command line control of everything,
    alternate audio/video in/out.  Highly recommended.

MEncoder : video encode/decode partner to MPlayer.  I've had more success running mplayer and x264 manually
    than using this.  Still I can't complain about MEncoder from the command line.

MPUI : GUI for MPlayer.  This is horrific malware.  When you install it, it takes over your system without
    asking.  They do provide some tools for you to change this after the fact, but still should be avoided.
    Use Media Player Classic or VLC.

AviSynth : script thing to pipe video to other programs that read AVS scripts.  Dear lord.

Basically I've found that all the GUI's are broken, and all the video containers (AVI,MP4,MKV) are broken. The thing I've finally discovered that actually works is using MPlayer and X264 from the command line, and only working with split frames. Trying to work with video containers caused me all kinds of hurt because so many of these apps fail to unbox the containers right and screw up the frame rate or drop frames or other mistakes. Instead now if I want to work on a video I use MPlayer to convert it to raw frames.

mplayer -benchmark -ao null -vo png:z=5 video.avi

to dump frames to PNG

mplayer -benchmark -ao null -vo yuv4mpeg:file=test.y4m video.avi

to dump the video to YUV4MPEG format in "test.y4m" for input to x264

x264.exe --bitrate 10000 --output "out.mp4" test.y4m

x264 compress to "out.mp4"

Then use mp4box to put the audio back if wanted.

The cool thing about mplayer is that its audio/video decoders are the same ones used to view the video. So you can watch it, and if it plays right in the viewer, then it will extract correctly. I've found lots of videos that I can watch in MPC or VLC, but then fail to load the same way in whatever encoder/decoder when I try to process something.

The sucky thing about this method is you make ginormous temp files on your disk, which also slows things down a lot. But avoiding the fuckups of going through the borked DShow codecs and splitters is worth it.

Most of these tools now are originally Linux tools that are getting moved back to Windows. One very promising development is that many of them have the option to directly load libs for the codecs Linux-style (eg. just load libavc to play video) and avoid DirectShow completely. I haven't really tried that yet but it seems like it's almost possible to work with video just by manually picking a few of these libs and then you avoid the whole Windows borked media layer.

ADDENDUM : one of the difficulties I've seen in a lot of tools is reading the frame rate wrong. This is presumably due to the demuxers not reporting things back totally right. But there are also two related fundamental problems that make everything harder :

1. Most of these formats don't have real/useful headers. (if they do have header info, it's just added as "comment" information). This was done originally because theoretically if your AVI is being broadcast on TV and you change the channel into it, you will just start getting bytes in the middle and never see the header, thus they didn't put good headers on at all. This is fucking retarded.

2. It's almost impossible to really reliably just get the frames out of video. DirectShow doesn't have a reliable call that's just "give me the next frame". Instead you have to ask for "when is the next frame" and then "give me an image at this time". The problem is that the "when" can get fucked up in various ways, and then when you say "give me an image at this time" you can either skip frames or get duplicate frames. (this is what fucks up the MSU VMT tool for me so badly, they are getting the time sampling all wrong quite often).

Even if it's not way off, this still causes subtle bugs because people don't agree exactly on how to represent the frame rate. Some people treat broadcast as exactly = 30000/1001 fps and use rational arithmetic for all timing. Some people use floats for frame rate and use 29.97003 and then wind up with floating point precision problems at high frame numbers. Many of the containers store the frame rate as a number of microseconds between frames, eg. 33367 ; so if they store "33367" in the header, should I use that as my frame time increment exactly, or should I use 33366.666666 ?

I'm guessing that tons of people get duplicate and/or dropped frames because of this and just don't notice it.

11-03-09 | Notes

WD-40 is not a lubricant. It is a solvent.

Ooo ! Ooo ! I want Google Voice really bad, just for this :

The SMS messages that are sent to your Google number will also be
displayed on the website. You can reply to the SMS from the Web as well.

holy christ it's so retarded that I get texts in my phone and have to reply using my phone keypad when I'm sitting right here at my computer. All forms of my communications should be accessible from any kind of electronic portal.

I pray to god Google puts the predatory incompetent corrupt cell phone companies out of business. Maybe next Google will open a bank?

Actually a Google Bank is not a terrible idea.

... well I got in the Beta for Voice. Hmm.. I'm not sure that it's actually awesome. The problem is that the "read" status for voice mail & text messages is not propagated among the various devices (handset and computer), which means I have to look through things in both locations. That's annoying. Also the texts that are sent through to your phone show up as being from some random Google number, not the original sender, so you can't easily identify the sender in your phone, you have to read them all.

The transcribed voicemail is awesome, but again the result is that my real phone voice mailbox fills up with shit I've already seen on the web, and I have to go through and maintain that all the time. I guess if I had an Android phone and it was actually all kept in sync properly that would rock.

10-28-09 | ...

It is disappointing that we appear to have revived the sensual excess of Rome in the realm of food, but we have neglected sex and violence (orgies and gladiators).

10-28-09 | Diversions

We're going to SF for Halloween. It's weird to be going back; I've been thinking about what one thing I really miss and what I'd like to do, and I can't really think of anything specific. I just liked being there, wandering around, exploring, taking little pointless trips around the city.

San Francisco for Halloween is kind of a weird thing because so many typical costumes are just normal clothes there. Dress as a guy from the 70's like Starsky or Huggy Bear? Oh, that's just a normal hipster. Dress as Tobias Funke as in daddy likes leather? Just a leather daddy from the Castro. Dress as a woman? Just another tranny. It's hard to dress up as something unusual when everyone is so unusual all the time.

Starsky and Hutch great photo.

Crafty Goths really are fucking crafty with leather. Wow.

Speaking of leather crafts ; Mr. S Leather is really fucking disturbing. I don't understand at all the urge to do those things to yourself, or to see other people do them. I have a pretty high tolerance to raunch and this almost made me vomit. (there's nothing bad on the greeting page until you click "enter")

Sunrise / Sunset times in Seattle ; our days are now down to 10 hours and we lose 3 minutes every day. (of course even when it is "day light" it's fucking gray).

Superbomba! photo tumblr; pretty rad.

Little Red Studio Erotic Haunted House in Seattle looks so awkward and unpleasant. I really want someone else to go so I can hear about it, though.

I've had this dream for a little while of adapting a bicycle to ride train tracks by putting an outrigger on it. Well of course someone's done that which is rad and all but sort of reduces my excitement about the idea.

Pretty good historical costume rental shop in Seattle . I don't really like dressing up for Halloween, but I've always wanted to go to a real fancy dress costume party. But not like the ones that actually exist. The ones that exist are either fucking fancy society events where people dress nice (good) but everyone is a bore (bad), or house parties where the costume theme is "pimps n' hos" or some shit and people just get drunk and act like they always do (it's always funny when some college house party has a theme like that and most people don't even dress up at all, but there's always one girl who shows up very nearly nude). In my imagination I'd go to masked balls for young beautiful sophisticated people where the costume gives you anonymity and lets you sink into sensual depravity. Which leads me to -

Venetian carnival masks ; this one is apparently made by the Boldrin brothers who made the actual masks for Eyes Wide Shut. (BTW Scorcese really likes Eyes Wide Shut which prooves he has terrible taste). I like the historical myth of the Bauta, that it let upper class Venetians go incognito and cavort without staining their honor.

I finally made it to the Seattle Graffiti Wall with N ( see also ). It's a pretty rad thing. There were three guys there painting when we stopped by. It gets painted and repainted over and over; when we were there one guy was finishing up this really wicked octopus, but it's probably gone by now.

Afrikando Afrikando is a pretty fucking great experience if you like weirdness and adventure dining. There's a lot of fun little ethnic pockets down in South Seattle.

Taqueria La Estacion in Burien are the best tacos in Seattle. If you're driving around the Mexican part of town trying to pick which Taqueria to stop at, do what I do : look for a Mexican in a cowboy hat. If you just go to the place that has a bunch of vatos, it will probably just be cheap and not necessarilly good (and they're probably ordering tortas, not tacos); look for the well dressed older Mexicans.

10-27-09 | The Evil of Obama

The really tragic thing about Obama is that he's an illusion, a palliative. Back during the GWB years we had a brief moment where liberals in America actually paid attention to politics again and were fired up enough to actually vote and counteract the right-wing nutjobs who seem to always be fired up enough to vote. All that energy got channeled into electing Obama, which it turns out was just a diversion, a waste of energy. It's just like sporting events or booze or television, a distraction to keep the rabble from realizing the reality and rising up.

Liberals are self-doubting, try to compromise, try to be reasonable, while conservatives seem to just dig in farther, toe the line, and move further to the right. And then liberals compromise again. So it has been since the 70's. I was telling Jeff the other day, I think modern democrats are obviously much more conservative than Nixon (who actually enacted many of our modern social programs). Even with democrats in power in government it doesn't seem to matter, we've continued to move to the right during this term. You may say "no no we're to the left of where we were under GWB". Not so - the pendulum is currently in its left-most part of its swing, but the center of the pendulum swing is still shifting right.

Obama & the democrats so far have been extremely right wing in everything they've actually done. Some of the speeches sound somewhat liberal, but those don't turn into actual actions, and instead we get sly under the radar extremely conservative actions. The worst things that Obama has done have been the furthering of opacity of the executive office, continued extradition of suspects to countries that torture, continued domestic surveillance, and complete subsidy of the finance industry without any increase of regulation.

What we needed after GWB was a really big correction back towards civil liberty, transparency, higher taxes on the rich & corporations, real health care reform, real environmental rules, an end to corrupt subsidies for oil companies, natural gas, ethanol, agribusiness, huge cuts of the pentagon budget, a real new policy on prisoners, and we've gotten none of that. But the real evil is that despite all that, most liberals are still self-satisfied because we "did a good thing" electing a black man. Bully for fucking us.

10-25-09 | Seattle Election

In case you haven't yet, how to vote :

Initiative 1033 : NO NO NO. These sort of funding freeze measures are super retarded and are the reason why so many local goverments around the country are broke and cutting crucial services.

Referendum 71 : Yes. Domestic partner rights. Meh.

County Executive : Dow Constantine. This one's pretty important. Hutchison is the Sarah Palin of Seattle.

Mayor : meh, I think they're both pretty retarded clowns and don't actually have that much power anyway. I voted McGinn just because Mallahan's entire platform is absolutely nothing but political platitudes and mumbo-jumbo like "we need to make seattle a great place for seniors, and encourage diverse business opportunites". Okay ass-clown, way to be vague and promise nothing, you don't get a vote.

BTW this stupid issue of the Viaduct replacement has dominated local elections recently even though it's already been decided and is basically a foregone conclusion that we're getting a stupid expensive tunnel that will take forever. I think pretty much everyone has the wrong solution on it. The real solution is no tunnel and no fucking raised viaduct either. The whole problem is that the current viaduct is not earth-quake safe and we need to replace it. Well fuck replacing it; just tear it down. We don't need all these freeways and cars. Tear it down, make 1st street and the whole Pike Market area pedestrian-only, and use the money for some public transit. We would have a much better city that way. (it's basically what San Fran did with the Embarcadero after the quake which was a huge success for them).

Attorney : Holmes. I'm pretty meh on this but Carr is so anti-nightlife, all the music venues are very strongly against him.

Most of the council positions I don't know anything about, but I have heard Robert Rosencrantz on the radio a few times and he is pure evil; he's a landlord & real estate investment advisor and basically wants to fuck over everyone but landlords and real estate speculators. So Position 8 vote Mike O'Brien.

These kind of non-partisan council elections are a big problem because the council is what actually has the power in Seattle but voters don't follow who's who. I really think that our democracy would work better if you just voted by party or something. Instead we have "nonpartisan" elections where voters are supposed to inform themselves (lol) and vote based on the issues (lol) as opposed to listening to unrealistic promises or just judging people by their looks and charisma.

BTW I think it's fucking retarded and probably illegal that you have to put a stamp on your damn ballot to mail it in. It should be prepaid.

10-22-09 | Inaction

There's a mistake on my cable bill. They're charging me $5/ month for cable modem rental, when I had my own cable modem and not renting. Calling them to fix it will take at least 30 minutes and may or may not involve some aggravation. I'm just not gonna bother and pay the damn $5/month. Maybe if I have some other reason to call them someday I'll fix it then.

I broke my TiVo a month ago or so. I was trying to replace the fan, which I did, but now whenever I boot it freezes up. If I do the status-46 key code thing, it tells me it detected a temperature too high and shut itself down. I think that's spurious, there's no way it's too hot, something is busted in the temperature control. Whatever, I'm just not gonna fix it. I'm not watching TV live or on DVR any more at all. I get torrents for shows occasionally, mostly I just download whole seasons.

I've written before about how my damn Nokia 1208 cell phone randomly doesn't send texts. Obviously it's a bug in the firmware which this guy describes decently . There is a firmware update which probably fixes it (Nokia is totally mum about the bug and whether they fixed it because they're fucking cunts who don't want to admit a bug because it's bad publicity). It's totally not worth trying to flash my fucking phone firmware. It was free after rebates and I can get another newer phone for like $10 after rebates. I'll just live with the bug until I get a new one.

I sort of feel like if my laptop or car broke, I just wouldn't replace them and I'd quit driving or computing or whatever.

Why do visually striking movies have to be such fucking pieces of turd? Can't a director make beautiful images and also make the movie not suck balls? We just watched "The Fall" which is just unbearable; yes, it's occasionally beautiful (the beauty if basically just shots of India, but seeing India well framed is nice). But you could say the same of "The Cell" or "In Dreams" or "The Science of Sleep" or etc. etc. I guess when a movie is incredibly beautiful and also has a tolerable plot, it's a rare and delightful pleasure. I'm also just kind of sick of the flashy high-saturation "beauty" that has characters dressed up in bright red flowing cloth standing in the middle of a desert with cameras flying around them.

I'd like to have torrents that run at full speed when I'm doing nothing else on the net, but automatically shut off when I'm trying to do something. And the torrent server is a separate machine from my laptop so it can't be software that just runs on my machine, I guess it has to be in the router. Does a router that does "QoS" actually do this right?

One of the interesting articles in the IEEE magazine recently was on "Flow" streaming routers .

I'm pretty annoyed with my HTPC. It's annoying enough that I wind up not watching TV or listening to music because I don't want to deal with it. There's nothing wrong with the software or hardware, it's just inherently a bad thing I now realize. I like having the music as physical objects. I want to be able to set my favorite music on top of my music player. I want to move them around the room. I want to be able to browse my music collection at the same time that I have something on TV - this it turns out is a major annoyance of the HTPC. In theory you can still watch shows and browse music at the same time, but in practice it just sucks to have a single device that controls both. And having your TV as the interface to your music sort of sucks too, it means you have to turn on the TV to play music, which is not cool when you're trying to lounge around on your bear skin rug with scented candles lit and whatnot.

I also keep forgetting that it's just a PC. Like I still can't get the fucking MediaPortal Netflix plugin to work right. But you know that doesn't really matter, I can just use the mouse to go to Netflix through IE and just play movies that way, I don't need the damn MediaPortal integration.

I also get annoyed by making playlists with the remote control; it's just too hard to browse through a giant music collection to find things and assemble a playlist. But of course it's just a computer! I can browse the shared music folder from my laptop, make playlists, and save them to the HTPC, which auto-loads them. (the only bit of a fudge there is WinAmp saves playslist as relative paths, so I have run a script to make them into absolute paths; no biggie).

Anyway, I guess I'm stuck with it, there's not really a reasonable alternative. I've been thinking now that I only use the computer to watch things, I could just have a monitor instead of a TV. I'm not really clear on what the difference is between an "LCD TV" and an LCD monitor. It seems like the TV's are just generally worse in terms of contrast and black level and response times.

Watched "Sin Nombre". It starts out pretty promising; I like the story of the immigrant family and the extra silly fictionalized action element from the violent gang to spice things up. But then it degenerates into "girl being a fucking stupid cunt" story element which I find so intolerable. This is a standard narrative crutch in which writers create tension and danger by making the female characters just do completely retarded selfish emotional stupid shitty things.

10-21-09 | Buying A Bike

A lot of people have been asking me for advice on buying a bike recently, so I thought I'd write a few notes.

First of all, if you aren't currently riding much at all, then my main advice is just to buy something cheap and plan to replace it in 6 months. If you haven't ridden real road bikes much at all, they will all feel weird to you and you won't know how to tell if they fit, so you will most likely make a bad purchase. Try to buy something for $500 or so and don't sweat it too much and then replace it when you know better. When making this initial cheapo purchase, worry more about fit and less about materials or components or brand name. The most important thing is to get your body used to the feel of a proper road bike.

Secondly, you have to decide what you want this bike for. Most of the bikes in a bike shop will be "road bikes" that are targetted at racing. This is probably not what you want so you should ignore all these bikes. These bikes are usually very light, very stiff, very jittery, with tiny hard saddles, very low handlebars, no room for fenders or racks, with very aggressive geometry that requires you to be riding hard and leaning forward a lot. You probably want a bike to get around town, that's pretty fast, can handle some wet road, maybe a rack to carry things, that you could ride out on the weekend for exercise once in a while. In that case what you want is going to be a "commuter" or "touring" bike (note, NOT "hybrid" bikes or bikes with suspension). They should have a slightly more relaxed geometry, room and braze-on bolt holes for fenders/rack.

X. Saddle. The main issue here is that most "road" bikes come with saddles that are too small and hard for the way people will ride them. In particular, a correct saddle is a personal thing, they should be sized to your sit bones, you may or may not want a taint-saver cutout (I recommend them), and if you are not going to ride with padded shorts, then you should have a semi-soft saddle. (note : NOT one of those big honking "comfort" saddles, but a race saddle with small gel pads or something like that).

A really good bike shop will let you swap the saddle. Most won't. If you can't find a bike with a saddle you like, just replace it.

X. Stem / handlebars. This is one of the crucial fit adjustments. (I don't mention the seat post, because pretty much all bikes have enough seat post adjustability to fit any reasonable rider). Lots of stems now are the threadless kind without much range of adjustment. IMO that sucks really bad. It means that you need to spend more time in the purchase making sure that the stem/handlebars are in a good position for you. If possible buy something with the ability to do a little adjusting. In our modern era you probably won't be able to avoid having to buy stem extenders or a flippable step or something similar.

In particular, if the bike stem is already angled way up (as many bikes sold today are), then you won't have any ability to adjust up more. That means it should feel plenty high to you as it is. As you get more experienced you can buy another stem that's flatter to bring the bars down some. In addition to the bars moving up and down think about how you might want to move them forward to back.

X. Tires. People spend all this money on bikes and worry about what the frame's made out of blah blah blah which doesn't hardly affect your ride at all. The #1 biggest bang for your buck thing that affects your ride is tires. Most "road" bikes are sold with tires that are too thin and hard for what people use them for. Conversely, if you want a faster bike, you can get a huge bump by taking that old beater and just putting some thin hard tires on it. It's a lot like swapping snow tires for racing slicks on your car. There are basically two extremes (ignoring mountain bikes / cruisers / etc) :

Commuting / rough roads : you want something like 28c or 32c tire , with some real pattern on the tread so it can grip in sand and wet. Inflate to 60-100 PSI. The wider tire is a bit slower, but will grip a lot better when you go over rocks and sand and pot holes. The lower inflation smooths out road bumps and also provides better grip. (It's an old trick that in extremely bad grip scenarios you can deflate even more, but that slows you down a lot).

Speeding : on good roads, you can run a 23c tire at 120-140 PSI with slick or semi-slick tread. This gives you very low rolling resistance, you can just touch the pedals and the bike glides and feels like it will never stop. If your bike is in good kip, there should be almost no friction in the turning parts of the bike, it all comes from the tires on the road. Of course these tires are very dangerous in the wet or even on sand. The hard inflation also makes the ride very rough, and can make the bike feel "jittery" because every little pebble is transmitted to your hands.

Often people will test ride some racing bike and think "wow that's fast" when in really all they're feeling is properly inflated fast tires.

X. Some bikes I like :

Navara Randonee at REI
Jamis Aurora
Salsa Casseroll
Surly Cross Check
Surly Long Haul Trucker

These are all around $1000 sadly. Most of these are classified as "touring" or "cyclocross" now. In reality they're closer to old fashioned "road bikes" than the modern compact little racing bikes. An old road bike like a Raleigh or all the Japanese bikes had braze on holes for fenders, racks, had relaxed geometry for comfort on longer rides, had some more room for adjustment of the stem/handlebars, etc.

ADDENDUM : buying used can be a great deal, but you have to be pretty smart about the condition of the bike and pricing. Some people have very crazy ideas about what things are worth. They'll say "I bought it for $600 ten years ago, so I'm selling for $550" . Umm.. no. A used bike should be half or less the price of a new one generally. If you look around you should be able to get bikes like the above for under $500. Also, you should not buy anything very old. Even if it works great it will be a pain to maintain or upgrade because the components are not compatible with modern gear. The oldest you should go is early 90's , and you want Shimano kit (though the chance of getting Campy on a cheap bike is very slim, you don't want it because replacement parts are so much more expensive and hard to find). Some people have absolutely insane ideas about what very old bikes are worth.

For your first time buying a bike, you can get a decent approximation of your size just by taking your inseam * 2/3. So for example my inseam is 34" * 2/3 = 57 cm ; I actually ride a 58 or 59, but it's close. I don't recommend sizing yourself this way for a serious purchase, you should try the bike and feel it because bikes with different geometry can have nominally the same "size" ("size" = seat tube center-to-top length), but it's an okay way to size yourself to find a very cheap craigslist first bike.

10-21-09 | Deadwood

I'm watching Deadwood, I'm into Season 3 now. It's good and all, but :

It's ridiculously Shakespearean. I mean it's literally a Shakespeare play staged in the old west; you have the Fool (Calamity Jane), the Narrator (Farnum), the tragic hero, the plotting sinister antiheros, the lower class righteousness and anti-government, and most of all you have the long soliloquies where people go off on these long explanations of what other characters are thinking and hypotheticals and so on. The monologes by Swearingen and Farnum are particularly hillarious.

"Yond Bullock has a lean and hungry look; He thinks too much: such men are dangerous". (Act I, Scene II). "Men at some time are masters of their fates: The fault, dear Farnum, is not in our stars, but in ourselves, that we are underlings" "If you prick us, do we not bleed? if you tickle us, do we not laugh? if you poison us, do we not die? and if you wrong us, shall we not revenge?"

I just completely randomly opened some Shakespeare and got this :


K. John. Though you and all the kings of Christendom Are led so
grossly by this meddling priest, Dreading the curse that money may buy
out ; And by the merit of vile gold, dross, dust, Purchase corrupted
pardon of a man, Who in that sale sells pardon from himself, Though you
and all the rest so grossly led This juggling witchcraft with revenue
cherish, Yet I alone, alone do me oppose 170

which is straight out of Deadwood. Anyway, it's so obvious that it must be intentional by the creators. I haven't done any research into whether they admit it, maybe they do.

I find the anachronisms a bit disturbing, but since it's just a play I roll my eyes and let it slide.

I find the constant repetition of "cock-sucker" and "fuck" to be pretty tedious. It doesn't make the dialog more realistic or gritty. Are HBO viewers fucking time travellers from the 1950's where all it takes is a bit of profanity to get them all excited? If you want to do an old-west show where the characters are "salty" then make it more fucking interesting, use some actual period slang for one thing, and fucking vary it up so it's not just always the same shit. It's not funny, it's not interesting dialog texture, it doesn't add impact, it's just fucking juvenile.

Also the Calamity Jane actress/character is totally intolerable. Following my great success in just skipping past the horrible cooch girls in Carnivale I'm just skipping every scene with Jane in it. It's great! Not only do I get to skip over awful dragging bad parts of the show, it actually makes the rest of the show more interesting because I have to figure out what happened in the parts I missed. All TV these days is way too repetitive and leaves out all mystery for the viewer; they lay everything out so obviously that you don't have to use your brain at all to follow along; things are much better when they challenge your mind a bit to put the pieces together yourself. You can create this puzzle for yourself by just skipping over 25% of any TV show.

10-21-09 | Jesus fucking christ Gmail

Well I now I have to dig through my Spam folder every day and see what ridiculous thing gmail has got wrong.

Here's an example of the kind of thing that's getting spammed : (this is from my landlord who has a fucking GMAIL email account and who I've mailed with many times before). This is the entire text of the mail, so you tell me what fucking Bayesian model decided this was likely spam :

Thanks for reminding, they should be able to come in this week so I will
let you know which day and if you can't be there I will meet them

Sent from my iPhone

On Oct 19, 2009, at 8:22 PM, Charles Bloom wrote:
>    Hi Lisa, is something happening with the fireplace?

WTF !?!?!?

I mention the fact that it's from a gmail account because obviously gmail should be able to direct-send mail internally and validate that it actually is from that sender; in that case you should be able to just identify accounts as either "spammers" or "not spammers" ; it's not like I have friends who will occasionally send me spam that should be filtered; if someone is my friend you should always let their mail through regardless of what the content is, hell if one of my friends forwards me one of their spams about Cialis because it was funny, that should get through.

The converse is a little bit trickier, you can't necessarily exclude all mail from "spammers" since I've got some corporations marked as spammers but would still like some mail from them to get through. That's a more tricky edge condition type of thing that I wouldn't be surprised if gmail got wrong, but oddly it never seems to get those wrong! It only gets the incredibly obvious ones wrong.

10-21-09 | The Day's News

ADCTC is the best published image compressor. It's a pretty obvious jamming together of various old ideas. It uses DCTs of various sizes for transform, up to 128x128 and allows non-square partitions. It picks optimal partitions like an old "Wavelet Packet" coder. Then the DCT coefficients are coded very much like ECECOW using a context coder with big contexts (ADCTC doesn't actually code bit planes like ECECOW does, it codes significance and top bit position, but uses contexts similar to ECECOW; the older verison AGU is really just like ECECOW/EZDCT).

DLI appears to be the best image compressor now. He's released zero info about it. So fuck him.

Twitter image compression challenge is cute.

I made new compiles of the LZP exes (as a I posted about before) ; they should not crash now. Back in the Amiga days I would always write my string matchers without bounds checks, because reading out of bounds was not an access violation, and then I would clamp the length after the fact. That is, rather than something like :

for(i =0; i < maxLen && ptr1[i] == ptr2[i] ; i ++ ) ;

I would just do

for(i =0; ptr1[i] == ptr2[i] ; i ++ ) ;
i = MIN(i,maxLen);

(this is simple pseudo-code to make the point, not actual code).

That doesn't work any more now that reading outside of the allocation is forbidden (there are ways you could make this style of thing still work of course, by allocating extra space off the end, and then concatenating a sequence that does not occur in the original text; I'm not sure if there's a very simple/fast way to find a sequence that never occurs in the original text).

Anyway, this was because the Squeeze Chart guy tested the old LZP's ; it's kind of amazing that they're still vaguely competitive in terms of speed/quality, though the more modern variants of course do better (such as Yann Collet's "LZP2" and of course higher compression LZP variants like the stuff by Ilia Mureyev etc. from compression.ru)

In other news :

Said the Gramophone is a good music blog.

From Busto to Robusto is a pretty nice poker show about real internet poker. (2 Months 2 Million is a fucking clown show)

George Nakashima is a fine wood furniture maker from Seattle. In the 50's he did seminal work creating simple furniture which explored the natural grain and shapes in the wood, letting it speak for itself. It's rather old hat now as so many cheezy places like ABC / DWR copy the Nakashima style, but the originals still have a profoundly elegant combination of understated refinement and untamed natural forms.

10-19-09 | Against asynchronous GUIs

Most new Microsoft products have super asynchronous GUIs. This means they do GUI refreshes and window popups and threads and so on. This sounds like a nice thing, it removes big stalls and freezes in the GUIs and makes them "respond" more quickly, in the sense that when you request some new view it pops the window up immediately and then gradually fills it in, whereas before it would just freeze and wait for the new view to pop up.

In practice for power users this just sucks absolute balls. Power users use the keyboard. We use our peripherial vision. We don't sit around and wait for windows to pop up, we hit key combos that we know and we memorize sequences. So for example you might memorize that hitting Alt-F opens a find dialog and puts the focus in the entry window, so you can highlight the text you want to find, Ctrl-C, Alt-F, Ctrl-V, Enter, all in a row boom you're searching for that text.

BUT NO! No you aren't, instead you just pasted that text into your document because the fucking finder window was being built on a thread and wasn't opened in time to receive your ctrl-v so your ctrl-v went to the old window.

There are few obvious GUI design fails here. One is that asynchronicity and focus stealing should never go together. If a window opening is async then it should not transfer focus to that window. Also, if a key pops up a new window and transfers focus, do that immediately and queue up the key presses and don't process them until the action associated with each key press is complete. eg. if you get input like A,B,C, if you process A and it fires something async, then don't respond to B until the action from A is done if it's possible that they affect each other.

More generally, this programming pattern of finding clever complicated ways to hide the fact that your systems are overly bloated and slow is just not the win. You will only make the failure cases less common but more ugly. For example in this particular case I'd rather have a slow modal popup than an unpredictable async popup.

Now, there certainly is a place for asynchronicity in GUIs, namely in tasks spawned from GUIs that run in a pane. When you start some long process, that should be async from the GUI and run on its own thread, but I would argue that the entire GUI should run on one single thread and GUI drawing updates of things like icons and buttons should not be async (not any aspect of it that affects user input - purely visual updates could be async). So like for example, when you open a new folder, the population of the list for that folder should NOT be async because the user may be typing characters to navigate to names within the folder, and if you populate async you will screw up the response to their typing (it could be okay to be async here *if* you make it respond in the same way that it would if it was synchronous!). Ideally GUI operations should just be fast and not need to be async.

How do we not have a fucking universal laptop docking multi-port thing !? WTF.

The delay of Windows/PC's recognizing USB devices is really tilting. Every time I move my laptop or plug in the keyboard, it's wait .. wait .. wait .. okay now I can type. Urg.

I'm trying this new thing where I put my computer on automatic standby after 2 hours. I've tried lots of reminder apps before to make myself get up and stretch, but I always just ignore them. Standby is a hard enough obstacle that I think it might make me actually get up - if nothing else I'll get up in anger because the standby will fuck up what I'm doing. But when I come out of standby I have to deal with the fucking USB.

Computer-laid-out maps are abominations. Hand drawn maps are beautiful and convey so much more information. (what follows are just links related to maps, not examples of great hand-drawn maps)

Metskers Maps in Seattle is pretty great.

FeetFirst is a non-profit that supports walking in Seattle. Their maps are only okay.

See also : City of Seattle Map of public art and Historic Places in Seattle .

10-18-09 | Personal Whining

I'm reading Steppenwolf for some damn reason (fucking people recommending shit to me; it's almost a negative correlation between recommendations and what's actually good; any time someone is like "oh this is good, somebody said so" I now think it must be crap). Anyway, it's fucking awful, it's such self-indulgent mopey high school shit, oh I'm so intelligent and sensitive, I'm too good for this world, I'm above it all and it's trivialities and pettiness, god how juvenile.

Depression is fucking weakness, it's an excuse to avoid life. Irony and superiority are cowardice; acting like you're too good for something is just a way of avoiding trying it and failing.

Often I'll fail in some way to live up to my expectations of myself being strong and dynamic as I think I should be, and then I sink into a spiral of self-loathing for my weakness. Like when someone is being petty or insecure or hurt around me, and I know I should rise above it and give them what they want to hear, but instead I lash back at them, I should be better than that and then I hate myself for it. Then I sink into a wallowing pathetic mire of self-hate and I become quiet and morose and function even worse then, which makes me hate myself more for being such a coward. But all of that is really just a childish way to avoid trying, it's a way of fleeing the challenge, it's an excuse not to try any more.

It's very hard to tell sometimes what exactly is your personality and what is just some current behavior that you can and should change. Is my depressiveness and antisociality actually who I am, or is it just a cowardly behavior I've adopted that I could stop at any time?

Lately I've found it very hard to have any opinion about anything. I don't mean that I have no thoughts on it, it's that I can't figure out what opinion is really mine. The problem I'm having is that I try to be positive and see the good in things. Like hey, singing folk songs around a bon fire, that could be fun, you know camraderie, fresh air, the fun of making myself do something unnatural to myself, good right? Yeah, but that's not my real opinion, that's just me trying to be accepting. Actually it's just really awful, I hate it. But that's sort of just the sour side of me that thinks everything is awful, is that my real opinion? Like even all this electronic music that I listen to, I mean really (rational critical brain) it's fucking awful, and the whole scene is so unpleasant and cheesy and lame, but hey no (accepting see the bright side brain) , the people are silly and silliness is wonderful and it's just movement which is great. Anyway, I find a lot of the time when people ask my opinion on something my response is something like "oh yeah it's great! I mean really it's awful, but no, it's great!". To some extent there's also the "everything else is so fucking awful that in comparison it's great"; like we went to Lark recently which really if I'm honest was pretty fucking awful, but I mean really most restaurants are so absurdly bad that actually it was pretty damn good in the scheme of things.

Human conversation :

"This weather is something something".

"That sports team did that thing they did".

"What about that TV show that was funny? that was funny."

"Funny line repeated ad-nauseum out of context and made un-funny".

"Pleasantry asking about something in your life I don't care about".

"Opinion or speculation about something I know nothing about".

"Self-righteousness and rationalization of why my way is the right way".

There, what a nice chat we just had, now we all feel better.

I foresee my future : being a miserable fucking salary man, going to a job I don't particularly like because it's easy enough and pays well, taking as much time as possible on lunch breaks to avoid the drear of the office, talking to coworkers about banal bullshit, becoming more boring and bland every day, going home with no fire no energy no vitality nothing new to talk about for my wife, slumping on the couch and watching TV, sucking all the happiness out of our home, slowly killing our relationship, making her cry about what a dreary life she's in, while I begrudge her not having to go to work and don't think she's doing enough to reach out to me.

I just don't see how it could really be any other way, that's what being an adult is all about. Almost every adult male that I know has gone down that path or something similar, and it's certainly the direction my momentum is moving in at the moment.

The first part of a relationship is so magical, you just want to see each other all the time and it just feels so amazing to just sit and look into each other's eyes or whatever, but you have to be careful, because that inevitably ends; it's a limitted resource, it's a battery you're draining, or a burning candle. You can extend it a bit by burning it less often, seeing each other less, but that doesn't really work, it's more just a time expiration.

One trap is that it feels so good just to do nothing and be together, that you stop doing interesting things. That makes the battery of magic drain out much faster and you will quickly lose it.

Another trap is to abuse it too much just for the good feeling. It's like eating chocolate, or masturbating, or drugs, it's an easy mood fix. You had a bad day, you're down, you just want to see your lover because they'll make you feel alright. But no, don't use them like that, it's too easy, it drains the magic.

You can definitely prolong the magic by only seeing someone in special circumstances. Entertainers of course know this, they try to avoid breaking the veil. A great magician or entertainer won't let themselves be seen when they're not in costume and in character and "on" , with their charisma and energy pumped up. When they're preparing or tired or whatever they hide away to preserve the illusion. We humans are really stupid, we associate our feeling about someone with our experiences around them, so if you only hang out with someone when you're doing amazing things, you think they're amazing. While this certainly works it's not really viable for a long term relationship because you can't be "on" all the time and you want to see this person more often than that.

It's sad how fast you start to take someone for granted. Our human brains are only sensitive to deltas, so when you come upon a new situation you see the delta and think my god this is amazing it's so good it's so different, but quickly that becomes the norm and then you stop perceiving it. One trick I've found to rejuvenate it a bit is to spend time with other humans, and then you see how shitty they are in comparison and you miss your lover anew and appreciate how great they are.

10-16-09 | Computer Complaints

God damn it, it really bugs me that anything still goes through the Firefox downloader; I want "download them all" to completely replace the downloader, but so far as I know it's not possible. Anything you download manually is fine, the problem is so many fucking sites are too smart for their own britches and launch the download for you and then that goes to the standard downloader. If you try to right-click and download it manually you get some fucking php thing instead of the real file. URG.

Christ you gmail spam filter fuckers suck so bad. The constant stream of "Cialis ! Grow your Peener!" emails that were somehow not marked as spam seems to have abated of late, so kudos for that, but instead I'm now getting completely insane false positives, which is much much worse. Just randomly today I thought "hmm let's take a peak in my spam folder" and WTFBBQ there are emails from my grandma and my girlfriend in the spam folder which have absolutely zero characteristics of spam but gmail decided I didn't deserve to receive.

What's worse than the fact that these emails have absolutely no spam similarity in their content is that they are with addresses from whom I have received 100+ emails and not marked any of them as spam and now you decide that you know better than me. WTF that's such an obvious fucking early out "hey this sender address is obviously trusted, just pass it through". Yes yes I know I can manually create my own filters to do that now but I never should've had to go searching around my spam filter for this shit.

So far as I can tell it still completely ignores your personal history of what you mark as spam or don't mark as spam. That's just insane. WTF you should just never ever never have a false positive from an address that you have communicated with many times in the past.

... AND fuck you some more. I just dug through some more and found falsely marked spam from Casey, Sharp, and Hook, all of whom I've had countless emails with and not marked as spam. I also was trying to have a technical discussion about H264 with a guy, and gmail in all its fuckitude randomly spammed about half of his emails !! So I got enough emails through that I thought I was getting replies when in fact I only saw half of them. WTF WTF WTF.

One of the things a good spam filter should have is the ability to show you why it marked a given mail as spam, and let you say "you were wrong about this".

Also, when I clear my spam folder it seems to delete permanently !?

When you remove an item from a project in VC 2005 it tries to fucking delete it from source control !? What crack are you all smoking !?

10-15-09 | Yoga and Government

I went to a Yoga class the other day. I keep trying to get into it periodically because every time I talk to people about my back problems they say "you should do yoga", so I feel like I have to do it just to exhaust my options. It sucks pretty bad in many ways, but I guess it was good for me. I can tell I used various postural muscles that I don't normally work. I'm pretty good about doing PT and my exercises and so on, but left to my own devices I fall into that human trap of just doing the things that feel good to me; it's good to have someone force me to do the things that just feel awkward and unpleasant and like they're not really good for me, but actually are.

On the down side, I hurt myself a bit because I tried a back bend. It's a freaking beginner class, way too many students, most super out of shape, and the dumb teacher first leads us into shoulder stands. I don't think *anyone* should ever do shoulder stands, even if you are fit and flexible enough to do them without hurting yourself, it's still not actually good for you to bend the neck forward that hard and put so much weight on your neck. In any case certainly beginners should never ever do them. So I spend like 5-10 minutes in fucking child's pose because of the fucking inappropriate shoulder stands where I'm just sitting there being annoyed, and then he leads us into back bends. I know I shouldn't try a back bend with all my problems but I'm sick of doing nothing so I go for it. And I hurt my neck. Yeah, dumb. Now I hurt.

I see trainers in the park sometimes doing those "boot camp" things, and they have a bunch of newbies doing straight leg lifts for abs, which is almost as bad. Shoulder stands are just horrifically bad for the neck, they squeeze the hell out of your discs. Straight leg lifts are horrifically bad for your low back unless you are already strong and flexible enough. Fucking personal trainers are so incompetent at their damn jobs.

That reminds me, I ordered a cocktail last night, and it's always such a harrowing process for me. I don't want to order something that the bartender isn't good at making, but it's hard to just ask them straight out because they often lie to save face, or get offended or think you're a snob or whatever. I want to ask "if I order a Collins will you use mix or make it from scratch?" "how much ice and soda do you think belongs in an old fashioned?" "do you know what a real daiquiri is? will you squeeze fresh lime juice?" it's like I have to give them a job interview before I can order a drink because you know, god forbid that a fucking bartender actually know how to make drinks, it's only like their job that they do every day. It would help if people would just admit that they suck and tell me what they can actually do. Like if the bartender would just tell me "look bub I can't really mix a drink to save my life, you should just order a beer" I would be like "cool, thanks" and I would be happy about it. One of the things that's always pissed me off is restaurants that actually have one or two really great dishes that they do well, and then like twenty other dishes on the menu that are absolute shit. That would be fine if they would just admit it, I'd be like cool, gimme those ones you do well. (or even better, just don't even put the shit on the menu; I'd love to see more places that had super tiny menus, like maybe 5 dishes total and all executed perfectly). Sadly in the real world if you ask the waiter what the best dishes are you will more often than not be led astray.

(aside : I once had an idea for a small restaurant; call it "Braise" or something like that. Everything would be cooked ahead, and we would only do stuff that was like a 4+ hour cook, stuff like braised short ribs, cassoulet, whole roast pigs, legs of lamb; different each night, sometimes maybe just one dish, like one night we just do huge batches of paella on an open fire and everyone gets a plate of paella - the point is to do stuff you can't do in normal restaurants because they have to be able to fire individual orders one by one, because we have such a limited menu we can put more time and money into each dish)

I don't really know what we hope to get out of Afghanistan, and I think sending more troops is a mistake. Politically it's a nightmare for Obama, because if he pulls troops out it will get worse there and he will be seen as causing that, but if he keeps going or puts in more troops he'll be seen as getting us stuck in a quagmire (since nothing good will come of it; there's no end point). The fact is Al Qaeda is long gone from Afghanistan already. The entire modern culture of Afghanistan is built on either corrupt tribal warlords or extreme Islam (Taliban) - no military action is going to change that, you can only push the people to one side or the other, there's very little middle ground of reasonable people. You can't ever defeat the Taliban when a big chunk of the population wants them there and they will come right back across the Pakistani border. The alternative to the Taliban is supporting the warlord factions, which is basically what we've done so far, which leads to antagonizing the population and creating more Taliban because they see the corruption and excesses of the government and have no faith in the democracy.

King County recently has been run by a temporary executive Kurt Triplett. The government around here is a huge fucking mess. The city, county, and state services all overlap and conflict. The cities also often choose to purchase services from the county or state rather than provide them themselves (stuff like fire, police, animal control, etc). There're no income taxes and there are lots of voter initiatives limiting how the governments can raise & spend money. The city of Seattle and King County are both run by councils, and the Mayor and County Executive act a lot like a national president in the sense that they "enforce the law" as passed by the councils, and they propose budgets to the councils, but the councils have to okay everything. It takes forever to get new funding for anything, because it has to be proposed by executives, approved by one or more local councils, then it often by law has to be approved by voters and/or the state government too.

I've been listening to local politics on the radio during drives lately. Triplett is pretty impressive; he wasn't elected and he's not seeking re-election, he's running King County in the interrim. He's intelligent, speaks directly to the issues, doesn't talk a lot of nonsense politics, and is being forthright about the situation, which is basically that King County has a massive budget defecit, and the retarded voters won't okay tax increases, and they also don't want service cuts. It's really encouraging to hear someone like that talk directly to the issues.

Then you get to listen to the candidates running for his spot - Hutchinson and Constantine. They both seem like pretty big toolbags who refuse to talk to the reality of the situation, they keep saying they are somehow going to save services and not raise taxes. However, there is a big difference between the two - Hutchinson is just obviously a moron who has no ideas and never says anything but a bunch of political nonsense and tries to hit irrelevant emotional keywords like "politics as usual" and "elections bought" and all that shit. She evades any questions about cutting services or says she won't, but also insists she won't raise taxes, and as soon as someone tries to pin her down she squirms and dodges. It's revolting. The fact that she's ahead (presumably just because she used to be on TV) is pretty fucking sickening.

10-13-09 | Linkage

Junk from the past two months :

Some music :

That Sound Hosted by Dave Cusick, the best new music. » Blog Archive » That Sound #60 - Psapp, Dimes, Asobi Seksu, +-, Shugo
YouTube - Raphael Saadiq - 100 Yard Dash - groovy new soul
YouTube - Night By Night - Chromeo - new Chromeo is always good
SeattleDNB Mix Archive
Mod Blog » Music For Riding High - just cuz the bicycle race photo is so great
Ctrl_Alt_Dlt on MySpace Music - Free Streaming MP3s, Pictures & Music Downloads
23five Live Archive - weird dj sets
Made Like a Tree Podcasts

Some visuals :

Galleries Architecture Abandoned Seattle Flour Mill Fubiz™
YouTube - The Mill - video in that same mill
Wojciech Burkot's Photo Galleries at pbase.com - some random photos with bikes
Yskira, The Skira Yearbook of World Architecture
vintage english bikes - bespoke frame makers ; awesome galleries
YouTube - The fastest human powered bicycle
YouTube - Quite - yes we can - gorgeous demo
YouTube - Performance - bicycle spoof rap video ; pretty awesome
YouTube - Angles morts - someone sent me this
RRE Photography - Ryan's photos
Flickr The Zoneplate Pool
Flickr sinosoidal's Photostream
Creep. - a set on Flickr - great photos of urban decay
CGarchitect.com Architectural 3D Awards - nice
Artforms of Nature • subblue

Some diversion :

_A Trip Inside Your Coffee Pot_ - really awesome detailed coffee brewing history
That Which Does Not Kill Me Makes Me Stranger - New York Times - old article about ultra-endurance athlete
Luna Park Seattle History
Life of a Pig - Seattle chef buys pig at birth, photo guide through its life and slaughter
Cheese or Font - so fucking rad - Cheese or Font !
Hideous Belltown - awesome
dschwaz - the title is a link, the post is a picture, fun
Gravel Beach - great blog about PNW beachology
Mountain ranges of the western U.S. - Geography - awesome geography overview of the west
Urban Exploration Resource Forum - US Pacific Northwest
The Physician and Sportsmedicine Knee Pain and Bicycling - good article on bicycle knee pain
The Personal Assistant Thread - something I often considered
REI Half Dome 2 HC Tent at REI.com - awesome tent
Pike Place Market Lunch Crawl - Mouthfuls - these guys ate through every stand in the Pike Market (forum)
Dark Natasha - Adult Gallery - so awesome
nations photo lab - good B&W photo printers
MUTEK News - good electronic music festival in montreal
largehearted boy a music and literature blog
Idle Words 07.2009 - decent blog
Degenerate Art Ensemble - Seattle group, pretty cool
Canon S90 - I'll buy one of these if they ever come out
Calendars - Printfree.com printable monthly, 2009
Brotherhood of b - abandon hope all ye who enter here
Amazon.com Ashley Book of Knots (9780385040259) Clifford Ashley Books - rad
CHDK Wiki - the Canon camera firmware hack thing for old Powershots

Some science :

Why Should Engineers and Scientists Be Worried About Color - about color maps for data visualization
What color is a blackbody - some pixel rgb values
VS .Net 2005, how to disable intellisense...
VR Seattle Blog – Panoramic VR Images of Seattle - boring but useful for environment maps
Visual C++ Team Blog IntelliSense History, Part 1
vision.middlebury.edufloweval - evaluation of optical flow algorithms for computer vision
Theora 1.1 Released Breaking Eggs And Making Omelettes - supposedly new Theora is getting better ?
The Anatomy of Reference Counting «   Bartosz Milewski’s Programming Cafe
Source Checkout - meshimport - Project Hosting on Google Code
Roundup Recent sketches on concurrency, data design and performance. - macton's posterous
Ranger - Nearest Neighbor Search in Higher Dimensions
PngSuite - the official set of PNG test images - to stress your PNG reader code
Micro-Rendering for Scalable, Parallel Final Gathering
Lost in the Triangles » Blog Archive » Encoding floats to RGBA – the final
Lost in the Triangles » Blog Archive » Compact Normal Storage for small g-buffers
Locality Sensitive Hashing (LSH) Home Page
Humus - Textures - Cubemaps - cubemaps are hard to find, here are some
HaukeHeibelGaussianDerivatives - nice images of the Gaussian derivates used in computer vision
Fractal Explorer Pixel Bender Plugin • subblue
Flicker Free Drawing in Windows - how to draw your shit right in Win32, good stuff
EnterTheSingularity Hierarchical Cone Tracing
Diary Of An x264 Developer » Why so many H.264 encoders are bad
Diary of an x264 Developer Test clips
Crytek GmbH Presentations - the new realtime GI stuff
Cris’s Image Analysis Blog » Blog Archive » Gaussian Filtering
codersnotes.com - Ramblings - Graphics - Papers, please - nice collection
CAVE Projects What is a Good Nearest Neighbors Algorithm for Finding Similar Patches in Images
Blob detection - Wikipedia, the free encyclopedia
Blind source separation and Independent component analysis.
ANN - Approximate Nearest Neighbor Library
Malwarebytes.org - best malware cleaner
A guide and tutorial on using ComboFix - hardcore computer fixer if malwarebytes don't do it

10-12-09 | Some Food Notes

The NYT magazine this week is one of the worst ever; I guess it's going linearly downward all the time so it's not surprising that each week seems to be worse than the last. Every article these days is either "duh" or "so" ("duh" = my god that's so fucking obvious why bother writing about it, or "so" = that may or may not be true but it's so fucking irrelevant or provincial that I care not). Anyway :

My god I am so fucking sick of hearing retards talk about food and diets. If you all actually want to eat healthy, it's fucking stonkeringly simple - just eat natural foods with a variety of nutrients, listen to your body, don't eat all the processed poisoned shit, don't eat out much. But you don't actually want to be healthy, you want to obsess over stupid shit like new research in fucking omega-tryptolic-beta-lythogens and fucking calorie deprivation for life extension and whatever other useless fucking irrelevant distraction you can use this week to keep from just doing the very simple and obvious thing that we all know is right.

A delicious soda for adults :

Fill glass with mineral water and ice. Real mineral water is key, it should have a nice bitter minerally flavor, not mild.

3 drops vanilla extract.

One half to one teaspoon of simple syrup, depending on your sweet tooth.


Optional : garnish with mint.

On Wine Aeration (is it hooey?) : We talked about this at cantinetta ; interestingly there now seems to be some debate :

Wine certainly contains lots of sensitive flavinoids that change under oxidation :


this straightdope article seems to repeat the old "common wisdom" that wine exposed to air changes flavor because it oxidizes :


however, some modern experts claim that is basically nonsense, that the reactions aren't fast enough, and alcohol and tannin are both anti-oxidants and preservatives that stop that process. They claim the only things you actually get out of aeration are evaporation of ethanol and sulfides , which lead you to conclude the wine is "smoother" (less alcohol) and less "harsh" (sulfur smell)



they leave no conclusion about whether something like the penny + decanter method would give you appreciable oxidation in a few minutes swirling.

Had a fun day taking Thatcher around town. I still have yet to see a single Salmon in the fish ladder even though I've gone a few times when it's supposed to be season.

Cafe Besalu in Ballard has some fantastic pastries. Crispy buttery, not too sweet, maybe the best in Seattle, but I'm holding out that judgement until I try them again. Sadly the coffee is just disgusting piss. My god. Ballard is gentrifying fast and is now the best neighborhood for bakeries. There's a new cute mini-neighborhood around 70th & 14th that features Honoray (Honore) bakery, Caprice Kitchen and the new pizza place Delancey.

Delancey is the pizzeria opened by Orangette which means I'm boycotting it. Oh your fucking food blog is so fucking precious. Let's see, Brandon works the wood fire, makes the dough, rolls the pizzas, travelled the country to figure out the dough, and you ... throw on some toppings that are widely considered boring and what are holding back the pizzas ? Good for you.

Fortunately Ballard also has the charmingly un-pretentious Veraci which is maybe the best pizza in Seattle. They also have a cool page about their ovens

A week ago we went to Cantinetta. Review : the pastas are very good, house made, well sauced, not overcooked like most fresh pasta; they're pretty straightforward, just very well balanced, very nicely salted. The apps were also very nice. The secondi (meats) were all pretty boring though decently skillfully made, I would skip them and just get apps/primi. It's a nice neighborhood italian place, but the insane long wait we had to endure was not really worth it. On the other hand, I guess it is miles ahead of any other italian place in Seattle, so maybe it is worth it. Desserts were terrible, like they should be ashamed, really fucking bad. Oh it's actually quite cheap by modern restaurant standards, if you avoid the crazy expensive wine list that is.

Harvest Vine : pretty fanastic Spanish small plate place (not tapas). I recommend sitting at the bar and going on a Wednesday or some night that's not too crowded. Order a few things, watch what the chefs are making for other people and ask them what it is and order what looks good. You will need a lot of plates, it's not cheap, expect to spend $100 per person. Avoid the cold plate appetizers, stick with the freshly cooked hot foods.

Black Bottle : the food is basically shit here. It's a Belltown popped collar cougar / sorority trash whore pickup scene. Avoid.

Spur : It was disappointing. Every plate has some gel or foam on it. I'm not inherently opposed to gels and foams, but these were generally badly made and often unnecessary; they had too much xantham gum or whatever they use, it make them too thick and sticky and tasteless. The food is all well designed, there are good interesting flavor combinations, but the execution and ingredients are just a little bit below the level they need to be, which makes everything a bit disappointing. We also ordered 6 plates and they brought 5 all at once and then realized they were firing too fast and held the last one (it's really hard to find a place that will give you a properly paced dining experience here).

On the plus side, the cocktails are amazing, like maybe the best in Seattle. They're interesting, all of them are slight twists on classic flavors, but not *too* interesting, the basic delicious booze flavor shines through (unlike Licorous which is very interesting but too weird, too much lavender and thyme and molasses and craziness at Licorous). The fried hominy snack was also out of this world.

Recommended for great drinks and a snack, not for dinner.

Oh, and the sous vide quail egg was good, but again not mind blowing. It basically comes out like a very nicely soft boiled egg, but perfectly evenly cooked through, so it is soft and custardy but cooked. I stand by my belief that sous vide is really a treat for the chef (because it makes things easier), not for the diner (because you can get the same results or better without the sous vide). (there's a funny flap in Seattle right now because the health department just realized that "sous vide" involves holding food right in the danger zone of temperatures for a very long time, which means it is technically illegal; to legally use sous vide a restaurant needs an additional industrial food processing permit, which of course none of them have).

jesus if you drink 10 cups of 16 oz decaf coffee you may get the same amount of caffeine as one 8 oz cup of regular. So if you drink 20X the volume you might get the same caffeine. Holy fucking news flash.

10-12-09 | The Great Recession

In Praise of the Recession

Most people are only looking at the bad side of the recession, but it's also doing a lot of good things. For one thing, I believe that the malfeasances of wall-street have been rather over-stated as the cause of the recession. Yes, they were the straw that broke the camel's back, but the fundamental problem with the US economy is structural and has been building for nearly 40 years : a steady loss of middle class jobs leading to fall in median income, coupled with growth of imports and consumption. We've basically all been living in a fantasy land house of cards where people can get no education (college degree = LOL), have absolutely no skills, do no useful work (being middle men = LOL), and yet live in ridiculous luxury. Well, perhaps that time is over, and that's neither a huge surprise nor a huge tragedy.

People write about the behavioral changes caused by the recession as if giving up ridiculous unnecessary consumption is some kind of crisis. My god I might not be able to buy a new car this year !? WTFBBQ. If you look at the behavorial changes that are being blamed on the recession, things like : eating out less, doing more work at home, not using gardners or nannies, buying less new things, sewing, repairing, kids seeing parents being domestic, reduced work hours - these are all positive steps towards a simpler more home oriented life that will benefit all of us in the long term.

These are all changes that we should be making anyway, because our natural resources and our population growth will not tolerate the continuation of our lifestyles of constant disposable consumption. But people don't respond well to gradual changes. Over all these years that income growth has not matched inflation, people should have been simplifying, cutting back, but of course they don't. They need a punch in the nose. People respond to shocks - in fact they generally over react. The general pattern of human behavior is : ignore obvious warnings, ignore obvious warnings, oh shit something happened, severely overreact. Well, IMO we needed a big correction, so yay.

I think most Americans still aren't facing up to the reality of our humbler future. They think of this recession as a temporary setback, and afterward they'll go back to being an unskilled "loan officer" in a local bank and make $200k a year. No, most of you won't. America has very little advantage over the rest of the world, and in many crucial areas (education, IT infrastructure, energy infrastructure) we are way behind. Yes, I believe Wall Street will be just fine and traders will go back to making megabucks, but that is one small area where we do have an advantage, and the rest of you will not benefit from that.

The future of America is even further income inequality than we have now, with an even smaller upper class. The lucky few at the top will still be massively successful, but most Americans will be worthless cogs that scrape by an existance by scooping up the scrabs that fall from the table of the rich.

As our country falls into the shitter, we upper class need to think about how we are going to preserve a nice place to live for ourselves, and stop worrying about the rabble - they are already lost.

One issue is the blight of fallow properties. Seattle has not been terribly hard hit by this recession, but even we suffer now from vacant lots and half-finished construction projects, empty condos, vacant homes, etc. These badly hurt the neighborhood. It seems pretty easy to fix, you simply make some "use it or lose it" laws like Brazil : empty vacant lots that sit undeveloped become city property and then are either converted to parks or auctioned. Empty apartments and condos and homes are forced to sell at auction (to someone who will actually live in them, not absentee owners for speculation).

Another issue is that all the cool broke artist kids are being driven out of desirable neighborhoods. In the future America in which only the top 1% can afford to live decent lives, we don't want to exclude the young beautiful fresh blood. The easiest way to fix this is to make subsidized housing for hot young people. We should construct and provide hip urban lofts as low income housing, but rather than the current low income qualifications, require applicants to submit nude photos and write a condescending music review for pitchfork.

Another big issue is health care - we need to make sure that the massive burden of keeping alive the rabble is not too much of a drain on the wealthy. In particular we need to find a way to change the expectations of the rabble so that the idea that they deserve to be completely healthy throughout their lives and kept alive as they get old is out of their heads. If someone poor in China gets a disease, they don't expect to get treatment, they hope it passes, and might ask the doctor for some drugs to ease their pain as they die. That's a massive competitive advantage that China has.

10-09-09 | Misc

I just rebuilt my old LZP exes with VC 2005. No code changes at all. (the last build was 98-99 , I think that was VC6) I was a little shocked at the file size difference :

w:\exe>ziplist lzpexes_old.zip
Archive:  lzpexes_old.zip
 Length    Date    Time    Name
 ------    ----    ----    ----
  53248  10-20-99  11:18   lzp1asm.exe
  45056  12-08-98  13:24   lzp1.exe
  65536  12-08-98  13:25   lzp2.exe
  45056  12-08-98  13:29   lzp3.exe
  41472  10-10-98  18:13   lzp3o2d.exe
  60857  10-10-98  18:13   lzp3o2.exe
 ------                    -------
 311225                    6 files

w:\exe>ziplist lzpexes.zip
Archive:  lzpexes.zip
 Length    Date    Time    Name
 ------    ----    ----    ----
  90112  10-12-09  13:07   lzp1.exe
 102400  10-12-09  13:07   lzp2.exe
 102400  10-12-09  13:07   lzp2d.exe
 217088  10-12-09  13:15   lzp3.exe
 217088  10-12-09  13:15   lzp3d.exe
 192512  10-12-09  13:15   lzp3o2.exe
 192512  10-12-09  13:15   lzp3o2d.exe
 ------                    -------
1114112                    7 files

One of the first ads in the IEEE magazine this month is for NVidia Tesla rack-mount server blades. I'm not sure who exactly is buying these GPU computers but apparently somebody is. They're obviously awesome in terms of flops per watt, which is the real important metric, but there aren't that many applications that can get a good win from them currently (almost all of those applications are some kind of finite element simulation).

It is interesting the way parallelism and GPU computing have brought us back to brute force. It's sort of like how doing custom clever hardware keeps getting beat by cheap off the shelf cobbled together parts, similarly doing custom clever software can now be easily beat by just doing brute force and going wide. For example, I've been looking at some fancy high-dimensional nearest neighbor acceleration structures such as approximate KD tree searches, BD trees (box decomposition) and VP trees (vantage point) (see the nice free packages ANN or Ranger ). They in fact do work great on single cores, but you can beat them easily by just doing dumb brute force and going super wide.

People are so selfish and unashamed. There's this weird intersection where Rainier Ave hits Jackson and 14th and turns into Boren that I go through most days on my way home. Yes, it's kind of weird there, I grant that. But I've seen three different times now some jackass gets confused and is in the wrong lane, like he's in the left lane and suddenly realizes he needs to turn right - and the jackass either slams on his brakes and puts on his turn signal and stops in the middle of flowing traffic, or just cuts right across dense traffic to make the turn anyway.

WTF. You fucked up, you got confused, you were in the wrong lane - you don't get to turn now ! Fucking go past and pull out of traffic and get your shit together and come back when you can make the turn without fucking stopping in the road. I just don't understand how someone can think that's okay. You fucked up, now you have to eat the pain yourself. You don't inflict your fuckup on the rest of the world. I miss turns myself from time to time, but I fucking keep going and circle back around, I don't make other people suffer for my mistake. It's fucking selfish and inconsiderate.

Similarly, if you are a fucking tard who sucks at parallel parking, don't try to do it on a busy street where you jam up a million people waiting for your dumb ass to pull in and out over and over while you fail to park. You should be ashamed of your suckitude and you should pull off the arterial route so you can park in privacy and not inconvenience us all.

10-06-09 | Automatic Timed Reset

File under : "Shit that's so obvious I've thought about it for a long time but not considered it worth writing" , sub-category "you all fucking suck so bad let me tell you how to do your jobs".

Lots of things should have automatic delayed resets. What I mean is various settings should be temporary and restore themselves to default or previous after some time elapses. You should of course be able to choose the time. You shouldn't have to remember to restore them when you're done with whatever temporary thing you wanted to do. Some examples :

Telling the radar detector to shut up. This should be a button that's like "shut up for 5 minutes" and you can push it again to add time. Instead because this button doesn't exist, you wind up just turning it off, and then you forget that you turned it off and suddenly you have a ticket.

Turning off the smoke alarm while you cook. Duh. This is a great example of how good user interface controls make a product *radically* more functional - not just a small improvement. In many places that I've lived, I've had to just completely disable the smoke detector near the kitchen because it's too sensitive and goes off any tiem I cook. That means it's not functioning at all. If it had a convenient button to say "shut up for 1 hour" I would hit that instead of disabling it, and it would actually provide a service.

EFT authorizations. When you give someone your bank routing number to be able to pull money out of your account it should clearly be a short-term access code that has an automatic expiration date on it, so you can say here's an access number that will last a year. That way after a year I don't have to remember to make sure they aren't still pulling money from me. (some kind of public key / private key thing here is also obvious; your bank account number should be a private key you never give out; you should give out a public key which people can use to send you "messages" (transactions))

WiFi access points. Obviously any time you hit "okay" to log in to some random wifi at a coffee shop or something that should have a timed lease on it that automatically expires. It should be "okay allow access and remember this for 1 hour".

etc. etc. basically any time you do something and think "oh gosh I'm gonna have to remember to undo that in an hour" it means your life is being shit on by bad usability designers.

10-05-09 | Body Fixing

I think I wrote this before (?) but it's been crystallizing in my head more. In the past few weeks I've been really busy and exhausted and have been slacking about maintaining my good body habits, and now my back is killing me and my scap is winging again and I'm a wreck. The thing is, I've kept doing the massage and PT for the most part, but I lost the most important piece of all : maintaining good body usage patterns throughout the day. With that preamble I give you :

The three step process to fixing musculo-structural problems in your body :

1. Tearing down. In this step you try to break your body out of the trap of the old bad habits. You may have pinched nerve pain which is causing you to hunch in weird ways - so first of all you need to ice it and take NSAIDs or whatever to just temporarily make that go away. You probably have tight ropey knotted muscles, so get massage. You probably need to stretch parts that are over-shortened.

This step is crucial, but it actually does nothing to fix the problem or make it go away. You are just trying to temporarily clean the slate so that you can build up again without the bad patterns from the past. If you don't use that clean slate to move on to step 2, it's completely temporary and pointless.

2. Building up. In this step you try to build new patterns and muscles that will hold your body in the right way. This is your Physical Therapy appointment, your home therapy, your exercise, your yoga, etc. This is an hour a day of making your body do the moves very consciously using the right muscles and the right posture and the right sequencing of muscles. If you just go through the moves without being mindful about *how* you are doing them, it's pointless. The point is not to get stronger, it's to retrain which muscles fire for certain motions. Even simple things like sitting up in a chair you need to train your body to do it with the right posture and muscles.

This requires a lot of concentration because if you stop being mindful for a moment, your body will fall back into old habits which it has learned over many years. You have to remember that muscle control and balance and posture and so on mostly go through your reflexive lizard brain, which is really dumb and is trained through repetition. By sitting at a desk have trained your lizard for many years, for many hours every day, to do exactly the wrong thing. If you stop focusing for a second, the lizard takes over and does it's thing. You can use your conscious mind to take over the muscles and override the lizard. You have to do that frequently enough that it starts to pick up the habit on its own. Doing the Physical Therapy once a day is only partly about strengthening, it's as much about the repetition which gives us :

3. Rehabituation : changing your habits. The most important step is adopting the new pattern in your daily life, every hour of every day. The hour a day of physical therapy is a start, but if you do an hour of PT and then go back to bad habits for the rest of the day you will make no progress. Again the PT is pointless unless you use it as a jumping off point to rehabituation : using the right muscles and doing the new body patterns every day for hours.

The real point of this whole process is #3. #1 and #2 are just steps to get to #3. If you don't put in the real effort to rehabituate and make the new patterns the norm in your daily life, then it's all for nought. A lot of people will be in pain and then just get massage, and of course there's no long term benefit. Maybe they'll take the next step and go to a PT and do the exercises, but then every day they go back to old patterns and gain there's no long term benefit. You have to do it all, and you have to really commit to #3 every single day.

10-03-09 | How to Ride a Bike

I've been meaning to write this for a while, but keep losing my motivation. Well Fucking Dave Moulton posted in my blog so I decided it was time. This is a guide mainly for beginners to correct what I believe are common mistakes. I know it's pointless, nobody who could benefit will read this and get it, but here you go anyway :

1. How to sit . Before you start trying to move you need to know how to sit. You should not be sitting on your ass like a ton of bricks. You are doing something physical - engage your muscles! The saddle (sic. "seat") of a bicycle should carry maybe 20% of your weight; your hands should get 20% and your legs should get 60%. I think the best way to feel how you should be sitting on a bike is actually to get off the bike and get into the position that a sprinter gets in at the start of the race : squat down with your legs apart, put your two hands in front of you, now extend your legs and arch your back - that's a biking position.

Many people incorrectly think that they need to sit upright for their "back". Nonsense. Upright is actually very hard on your back, and if you sit with your weight on the seat it means you are transmitting every bump in the road straight into your spine. It's actually much more natural for your spine to be arched. Imagine yourself like a cat, your spine is bent, it has springy energy, when a bump in the road shocks you it just flexes the bow of your body a bit, no big deal (a proper biking position is kind of like the cat back position in yoga). (upright bicycle sitting is good if you have a bad *neck* - the forward leaning bicycle position while actually good for the back is bad for the neck when you look upward with it).

Never ever lock your elbows or your knees and just rest with them locked. They should be slightly bent at all times. Your muscles should always be engaged so that you are a big arch of muscles - never resting on bones or joints. You should be light on your feet with tension in your legs and your weight on your feet, not your seat - you should be able to "post" out of your saddle like an English style horse rider at any time. The way you make your ride soft is not by buying a big padded seat (which is very bad for you) it's by posting out of your saddle and turning your body into a bow.

Obviously you can't sit correctly if your bike doesn't fit you. This is too big of a topic for to cover in depth here, but I'll say a few things. Many road bikes sold to the casual market have way too big of a drop from the saddle to the handlebars. For casual riders you actually want the bars and the saddle pretty close to equal height. Sadly most bike shops either sell ridiculous casual bikes that are way too upright or racing bikes that have way too much drop, it's hard to find a comfortable efficient road bike but it's getting a bit better these days.

(aside) : Another issue with fit that I'll mention because most shops fail on this so hard : presumably you are a beginner or a semi-amateur like myself that goes in an out of shape; if you are buying a bike you are probably not in great biking shape at the time that you buy it, but you will be riding yourself into shape over the course of a season. As you get into better shape, you will want a different position on the bike, a more aggressive drop, perhaps a more powerful saddle position. What this means is you need some room to adjust in your bike sizing. You should never buy a bike that just barely fits you, or where you have to put the seat all the way down to the frame to make it low enough, or all the way up to the end of the seatpost to make it tall enough. It's important that you get a frame that is comfortable now in a more relaxed position, but has room to adjust towards more aggressive and still fit you. In general if you're deciding between a frame that's a bit too big (so there's very little exposed seatpost) or too small (lots of exposed seatpost) it's better to go with the frame that's too small, because you do have some ability to make it "act bigger" through the stem and seatpost, but you can never get smaller than the frame limits.

The other big issue is how your saddle contacts your body. You should feel the contact on your sit bones. You should not be sitting on the nose of the saddle with it jammed into your taint. The nose of the saddle is not for sitting on, it's for guiding you on and off the saddle when you post in and out of the saddle. When you sit back onto the saddle you should feel your sit bones contact the two padded wings near the back of the saddle (but not so far back that your thighs run into the saddle when you pedal). Your pelvic bone should be pretty close to vertical - not tilted way forward so that you are jamming your gonads into the saddle. Your gonads should basically not be contacting the saddle at all; you'll hear beginners say "ow it hurts my balls to sit on those hard skinny race saddles" ; well dumbass you're not supposed to sit *on* your balls. When you lean forward to reach the handlebars it should come from arching your back, not rotating your hips - your pelvis should stay vertical. Note that saddles are not one size fits all ; you should get a saddle that is the correct width for your pelvis and sit bones (women in particular should usually have non-standard saddles, but so should men with unusually wide or narrow hips).

You need to get comfortable in "ready position". When coasting, get your pedals level with each other, stand up on them so your butt is just barely off the saddle. Hold your bars with both hands, elbows and knees should be bent, back should be arched. This is the position where you have the most control of the bike and are ready for anything. This is how you should go over bumps, this is how you should be when you might need to brake hard, when you're around dangerous traffic, etc. Any time you are riding you should be able to post up into ready position at any moment.

2. How to brake . This is perhaps the biggest mistake that I see beginners make and maybe the most important because good braking is critical to surviving.

You stop yourself with the front brake. To stop quickly, you need to pull it *hard*. You need to get comfortable with this before you go riding at all, because when that car or pedestrian jumps out in front of you, you should be able to stop from high speed fast. Lots of beginners go off riding fast and can't stop themselves and it's a huge mistake. You need to hold the handlebar hard while you brake, your hand should wrap around the brake and the handlebar and you squeeze them together.

Beginners are afraid of pulling the front brake hard because they are not in good control of their bike. First of all, learn to sit right and be comfortable in "ready position". Now, when you brake really hard, hold the handlebars firm with both hands so they can't twist. The bike is going to stop suddenly which is going to fling your body weight forward and try to throw you off over the front of the bike. If you are in ready position and using your muscles that's not really a problem, it's only bad if your body is limp or your limbs are locked and you're just sitting on the saddle. When you jam on your front brake, you should push forward with both your arms, sort of like a push-up movement, but to push your body weight backwards towards the back of the bike; this braces against the momentum that will try to send your body over the front; you also want to crouch down on your legs to lower your body weight.

You should practice stopping with only the front brake to make sure you are comfortable with it. In the dry you are not going to skid on your front tire ever - you will skid on the back very easily which is why you don't use it for hard stops. You should be able to stop from 20 mph in about 10 feet. If you can't do that, you need to practice jamming on your front brake harder. To brake really hard when going fast, you want to get your hands down in the front vertical part of the drops, get your weight up on your legs and crouch down and move your bodyweight back to get ready. Don't go fast unless you can brake properly when going fast.

3. How to pedal . Now that you know how to sit and brake you can start thinking about moving. The first issue is how to pedal.

First make sure your leg is moving through the proper range of motion. Your seat should be at the right height so that your leg can extend almost all the way in the down-stroke, but not so that your knee straightens completely or you have to reach with your tip toes or sway your hips. Your hips should stay level, your knee should stay bent at all times, and your foot should stay roughly flat the whole time (a tiny bit of "walking" motion is okay). Again one issue here is getting the bike sized right for you; to some extent you can adjust this by raising or lowering the seat if that works with the rest of the bike geometry; however one issue that is often not addressed is the length of pedal crank arms; people with shorter legs should have shorter cranks so that their knees are not coming up too high at the top of the stroke.

The next big issue is cadence (how often the pedals go around). You want to be pedaling at least 60 rpm generally. Amateurs often pedal very slow cadence in heavy gears and have to stand up in get their weight into it to turn the pedals over. You want your pedalling motion to be a circle, not pounding like pistons that only push in the downstroke. You want to feel like your feet are just constantly whooshing around in fast circles.

4. How to ride with cars . If you're riding in the city you need to know how to be a safe rider around cars. Your main concern should be your own safety.

The most important thing is being predictable and visible. There are a few things that amateurs do really wrong.

First of all, you should ride in straight lines as much as possible. This makes it clear the line you are taking. In particular, do not weave in and out of parked cars. A lot of amateurs are scared of traffic and want to stay as far to the right of the road as possible, so they will weave in to the right when there's a gap at a driveway or whatever. This is a very bad thing to do. Pick a consistent straight line and stick to it.

Second, claim your space. Make yourself visible and take the road when you need to. Don't worry about blocking cars if you have to. For example if you see up ahead that there's a branch in the bike lane or something, don't try to just barely skate around it and leave the lane for cars. Instead pick a safe time and pull out all the way into the middle of the car lane and claim the space visibly until you pass the branch. Don't wait until the last minute before an obstacle and swerve to avoid it - claim the space early and make it clear what line you will be taking.

Don't be afraid of the cars that are moving along beside you. They are actually the least of your worries (but do worry about them if they are braking or moving erratically as it indicated they are trying to parallel park through you or turn right across you). Do be afraid of parked cars - don't ride too close to parked cars or driveways. Generally being out in the middle of the road where you are visible and can see other moving cars is the safest area to be. Being near obstacles and things that block vision like buildings or parked cars is the worst place to be.

Remember to watch the road conditions and be vigilant and don't be afraid to defend yourself and inconvenience cars. Most of my worst crashes have been due to junk in the road that I hit because I was watching the cars, or I was trying to be polite to the cars and was riding in the shoulder or the gutter or something.

5. How to shift . I won't say much on this (see Sheldon on Shifting ) but two things that I see almost all beginners mess up :

First, bikes are designed for you to use the front & rear derailleur in tandem, not just one or the other. You should think of the chain as wanting to always be parallel to the bike. You want to shift such that the chain moves directly sideways either inwards towards the bike (lower, easier gears) or outwards (higher, harder gears). The chain does not want to be at an angle to the bike. So, you should use the small front chainring for the lower (larger) gears on the back, then as you shift up through the sprockets on the cassette (in the rear) at some point near the middle you should then pop up to the larger harder front chainring, then finish shifting up through the sprockets on the back until the chain is all the way to the outside.

Second, your front derailleur almost certainly has a "feathering" adjustment capability of some kind. What this means is that the movement of the front derailleur is not digital - it's not just on the small chainring or the big chainring in a fixed binary state. You can "feather" the front derailleur to move it slightly without shifting to the other state. This lets you move the derailleur cage so that it doesn't hit the chain. Any time you are "cross gearing" at all (small front chainring to small real sprocket or big-to-big) (cross gearing is bad, it's the angled chain that I said to avoid in the first section) you can use the feathering to keep the front derailleur cage from rubbing on the chain.

A properly adjusted bike should be almost silent. If your gears make a lot of noise as you shift, either you are doing something wrong or it is out of adjustment.

10-02-09 | Seattle

It's been one week of gray rainy shit and the winter blues have already kicked in for me. God it fucking sucks. There's this awful catch-22 I get stuck in. I'm depressed and I have no energy, so I don't exercise. But if I don't exercise I'll be depressed and have no energy. The bleck makes me want to just curl up on the couch and never move, but if I do that I'll be only more depressed. I desperately need to get outside and have some fun, but fun comes from the energy inside yourself, and I have none. When I can't exercise I can't sleep; when I can't sleep, I can't exercise. When I'm depressed I don't want to hang out with anyone because I don't feel like myself and I don't want to inflict my mopey ass on anyone; but not seeing anyone makes me depressed.

Of course it's a positive feedback loop cycle, but everyone knows how to get out of it - just by force of will you have to make the cart jump the tracks and hop back onto the good living line. Just fake it briefly with sheer will to paste on a smile and hopefully that takes long enough for you to get back in the good path for real. But I feel like I have to do that over and over, every week, or every day, slipping into depression and then fighting it and trying to kick start myself back into the good path, all winter long, and it's just exhausting.

I've been overeating recently. One of the ways you can tell you're overeating is if you are taking lots of giant shits. It's your body's way of telling you it doesn't need all the food you're cramming into it; it can't digest it all so it's letting some pass through. I generally have very good body awareness; I don't really diet, I just listen to my body, it tells me whether to eat or not. There's a problem with that though, when you aren't exercising or sleeping and you start overeating, then you feel all lethargic and have no energy and your body says "I have no energy, I need food" (and particularly sugar), so you eat more, but that just makes you more lethargic, so you want to eat more. It's another bad positive feedback loop.

I've been watching the first two seasons of Carnivale; it's fantastic. I think it's worth watching even though it was cancelled and it just ends. The setting and mood and everything are perfectly done. The Brother Justin character is just so well done, he's mesmerizing. There are some weaknesses : I think pretty much all the side characters in the carnival are badly acted and extraneous and feel just tacked on, they don't really integrate in the main story line. All the bits with the "cooch girls" are so bad and pointless that I actually started using the 30-second skip to get past them every time they came on. The bearded lady and the rest of the "freaks" are just tedious. Sofie and Jonesy are really terrible actors and are painful to watch every time they come on screen. Despite all that, the Hawkins-Justin storyline is so great and their characters are so perfectly done it's really worth seeing.

I find I'm renting movies from Netflix just to make it shut up about recommending them to me. There are certain things it keeps recommending to me that I don't really want to say "not interested" about because I might want to watch them some day, but I really don't want to watch them or even put them in my queue. But if I just ignore them, fucking Netflix will just keep showing them to me over and over as recommended movies. (of course I've written before about many issues of how retarded they are about what they choose to show you; in particular 1. stop recommending me movies that I ALREADY RENTED FROM YOU and 2. don't show me "related" movies that are highly correlated but in a completely different category, or at least give me a way to see only correlated movies that are similar by content) (I'd like to be able to click on a movie I like and find similar ones but it doesn't fucking work because of those two issues). The two biggest examples are Kurosawa movies and Ingmar Bergman movies. I don't really want to watch either of them at all, but I also don't want to say "not interested", so I finally just broke down and put some in my queue. Fucking netflix.

BTW yes I know about the new Taste Preferences thing and it's all wrong. You don't want to have permanent saved settings of what genres you like, on different days you might want different things. It should be easier to browse into the cloud of connections as a visitor and do your own guided exploration; you should be able to jump in using the movies you've seen before and then explore on axes of related people (same actors, same directors) , axes of genre (content / release date), axes of correlation to your history. Also they are way too detailed, but really anything saved like that is just entirely the wrong way to go.

I'm physically incapable of talking on the phone. When someone calls me I'm just like "ummm". My mind blanks, every word is a huge effort. I'm not sure what the problem is; I struggle with communication in general, and it makes me really uncomfortable to not be able to see the person I'm talking to. I'm also very visual and ADD and easily distracted, so I often find my mind has locked into something in the room that I can see and I have completely lost the thread of the phone conversation. Or maybe I'm just being a big baby like usual and if I would just "get over it" and try harder maybe there isn't actually a problem at all.

I'm so negative, I really don't like it, and I know it's no fun to be around. Nobody wants to hear criticism and ranting and whining, and it makes them defensive and afraid it will be directed at them next. I don't want to be so negative, but I just don't see how you can have a functioning brain and not think that 99% of this world is utter shit (of course 99% of you don't have a functioning brain).

For example : WTF is wrong with all you men who like gigantic fake boobs; seriously, WTF, you are retarded scum. Also WTF is up with all the Seattle drivers who love to go super slow, but then run red lights. WTF if you're in a hurry then maybe you should take off on the green faster and just get a fucking move on overall, and if you're not in a hurry then don't fucking try to sneak through the yellow. Almost every time I go through a yellow when I think "oops that was too late I shouldn't have done that" some jackass slow poke fuck-tard comes on through behind me.

Now that I know how easy/feasible backpacking is, it's the only way I want to experience nature. It's horrible how many people are around on the short day hikes and the car camping areas and national parks. There's really no reason at all not to backpack instead. By far the biggest problem is the weather. If I was in some place like my beloved wonderful California where the weather is predictably perfect all the time I could go often. (and in many parts of CA you could save weight by not even taking a tent)

I don't understand people who persist in chit-chatting about things they know nothing about in this modern era. There's this whole style of conversation that semi-intelligent people love to partake in where they conjecture about things they don't really know much about. I think they like to show how smart they are, how the can guess how a doodad works or the origin of a word or where some cultural practice came from. Ooh la di da nobody's fucking impressed. If you actually care to know about it, go fucking google it. If you don't care, then shut the hell up.

I've turned into a consumerist recently, where any time I feel like something would improve my life I just go buy it. I really don't like it, it makes me feel like yuppie materialist scum. I really enjoy repairing things and using my ingenuity to make do and so on, and I've lost that. The problem is like most things in life, it's really hard to find the sensible middle ground where you are making rational decisions. It's easy to swing completely one way or the other - either just buying everything and getting way too much junk that you don't really need and losing all your homemade joys, or being a martyr and buying nothing and making your life pointlessly harder than it needs to be. I've spent most of my life in the "martyr" mode, living without basic conveniences like pants without holes or DVD players or whatever.

Anyway, in attempt to make sensible use of money to improve life quality, I'm trying to spend to help me get through the winter. Last year I bought one of those fucking day spectrum light things that was a total waste of money. This year my idea is to get lots of warm dry clothing so that I can go out in the rain and feel good. I want fuzzy fluffy swaddling dry warmness. I'm trying to buy wool cycling tights but I can't find them anywhere (there are wool tights to be had but they all have loose crotches or flies which are not okay for cycling). I also

To try to cheer myself up during the commute I burned myself some CD's for cheezy sing-alongs in the car. I think U2's "The Joshua Tree" may be the best sing-along album of all time, it's got the ecstatic bursts where you just have to belt it out. The only problem with it is Bono's range is too good so he jumps around to notes that I can't hit. Another awesome one is Radiohead's "The Bends" (I also added the last two songs from Ok Computer ; Ok Computer was much celebrated but IMO the middle of that album just sucks balls, fucking karma police and fitter happier and just like lead weights of awfulness that sink the great momentum of that album; songs 1-3 and 11-12 are amazing and there's just this gulf of shit in the middle. Also #3 (Subterranean Homesick Alien) is probably the most disappointing song in the world. The first 30 seconds are just glorious beautiful unusual, they get me so high, and then it just turns into a boring old rock song after that; it's like they had this beautiful idea and then couldn't figure out what to do with it, so they were like "well just scream and bang on the drums and we'll call that a chorus") , but I digress, The Bends also has the problem that Thom Yorke sings too weird and high so I have to octave-shift to stay with him which is unsatisfying. Other cheesy sing-alongs : Bjork, Coldplay, Afghan Whigs, Neko Case.

Cbloom's perfect shoe is a tight leather sock. To put it on the first time, you have to soak it in warm water, oil up your feet, put it on and let it dry on your feet as you walk around so it molds to your shape perfectly. A good cowboy boot is kind of like that, but I want no hard sole at all. On the bottom of the leather sock there should be a very thin coating of sticky rubber (none of that hard vibram shit). There should be no support anywhere.

They should maybe be lace up with long laces that you wrap around the bottom of the foot like soccer shoes. Actually good soccer shoes are pretty close, but just with no sole and some glued on rubber instead. Laces that wrap around the arch like that provide all the support and tightness you need and feel really good. Maybe instead of a lace it should be a wide nylon strap.

I'm not really down with the design of most of the new fad of "barely there" shoes like the Five Fingers and such. I don't want toe separation, and I also don't want my feet open to the air because it's cold and wet out there. What I want is a shoe that doesn't enforce any structure on me, and also one that has almost no sole. Soles remove your ability to feel the ground and encourage bad walking patterns.

There are actually a lot of women's shoes that I think are okay; the ballet slipper inspired style that's fashionable these days is pretty okay, as is the very slim low-profile sneaker that they make for women. Sadly there are almost no shoes like this made for men.

"Five Fingers" are pretty close I guess, but there are a few problems : 1. I despise Vibram soles, the material is just no good, it's too stiff and smooth and not sticky; real rubber is the only way to go. 2. You don't actually want toe separation for various reasons (cold, fit, strength). 3. the strapping is insufficient to get them nice and tight; a Roman-sandal style wrap-around strapping is the way to go.

10-02-09 | Indulgence

There's this very disgusting type of indulgence that has become very popular in the last ten years or so. It's watching things that are forbidden to us that we secretly lust after, but watching people of a type other than ourselves do it so we can at the same time condescend and get the pleasure of feeling superior to them. I find it especially revolting because it's very dishonest. I guess people who watch things like "Desperate Housewives" get this pleasure; they feel "oh how trashy they are" but at the same the reason they like it is because they really want to be like that.

We love catching public figures having affairs because we can feel all superior ("how could they", "what sleazes") while at the same thinking "oh that intern is pretty hot, I'd like to tap that, way to go Dave".

Mad Men is one that particularly bothers me. The setting becomes an excuse to show mysoginistic chauvinistic racist behavior and it's okay because "that's how it was at the time". We the viewer can delight in feeling superior (oh look how they treated gays back then we're so much more advanced now) while also indulging our secret desires for repressing women and cheating and drinking and all the things we're not supposed to like.

I think a lot of smart white people who listen to rap are doing the same thing. On the one hand it's all very ironic and we can feel superior to the uneducated blacks and their classless worldview. But at the same time it's an excuse to secretly indulge in someone singing about sleeping around and treating women as objects with their booties and being all about the money and selfishness and your shiny grill and the benjamins and whatnot. It's sickening.

Dave Moulton is kind of a nut job, but every so often he posts some great stuff about old bikes he made or stuff about frame building in general ; he just posted this one which is one of the most beautiful bikes I've ever seen. It's pretty much how I wanted to build my new bike, but the practicalities of lights and water and fit get in the way (you may notice the drop from the seat to the bars on that bike is very high). (See also his gallery ; sadly it's some custom ass thing and not just plain HTML the way it should be )

When one of those pop-up fake virus scan things hits me in firefox these days, I don't try to close the box, it's too risky. Instead I have a batch file that does :

taskkill /IM firefox.exe

and I just run that to close the box. (also helpful when firefox does that awful "firefox is already running but not responding" nonsense ; if you can fucking tell that it's not responding then give me a button to kill it and launch a new one right there in the message box!).

That way you are sure to just kill the whole app and not accidentally hit okay on anything. I would *love* to be able to disable javascript and popups completely in my browser, but sadly it's not really practical any more. (In theory you can disable popups and then just allow them when you actually want them, but I find that breaks most web pages; eg. when you do go ahead and okay the popup, it comes up blank or missing data; I think it has to do with server-side popup generation, they get screwed up if they aren't allowed to do it right when they want to). (I find flashblock messes things up a lot too, when you finally allow it the page still doesn't work right, but that's still worth the pain for me).

10-01-09 | Excludes

Web sites and searches that you visit repeatedly need more memory. The simplest thing for them to do would be to save "excludes".

One example that really bugs me is eBay. eBay is a fucking nightmare to use because it doesn't let you save the filtering you do on one visit. Say you're trying to buy a certain type of bike. You're going to be visiting ebay once a day for a week or more checking out the listings. You spend a lot of time looking into the listings trying to figure out if it's a good one or not. Then you come back the next day and have to look through all the same shit !!! It's retarded. There should be a box that says "never show me this listing again" you can click (and it needs to apply when people relist the same thing, but that's a later step). You should also be able to exclude sellers, exclude by pattern match, move things to top, and save notes on listings.

The other place I really want excludes now is in google results. I do a lot of searching trying to find free download mp3's so I can listen to bands (I can't stand using the streaming media players on myspace or whatever, to check out a band I want to actually download some mp3's so I can listen to them without huge frustration). It's becoming very hard because there are so many awful sites masquerading as music content sites. If I could pattern match exclude them in the search results I could easily block out all the junk results.

I mean social search and whatnot would be nice and all, but I'd like to just have a decent way of social searching with my *self* first.

Also WTF the "t-type" word stuff in phones needs to use markov models for the whole sentence (so should the apple keyboard and all that kind of stuff); really it should be using PPMZ or a variant to make very accurate guesses of what letter you intend. It could also have a high confidence "complete word" so you can just tap a key and finish the whole word. The actual cases where there are ambiguity are very very rare, once it sees the whole sentence, usually only one of the possible words makes sense given the context.

10-01-09 | Memory

I was swimming at the Bellevue pool today and feeling semi-miserable and started thinking about a summer I spent in Austin. Just going to the pool and swimming for exercise for a while is really awful. One summer in college I stayed in Austin and did nothing. I basically had no friends or girlfriends all throughout college, and nobody was in town anyway, so I was all alone. It was glorious, I often think of that summer as one of the happiest times of my life.

I had a house in Travis Heights. It's a lovely residential neighborhood right near downtown with older single-story craftsman style houses; it reminds me a lot of Pasadena. The neighborhood also has a very friendly village feel, people have stickers about "Love 78741" or whatever. I got half a duplex for $500/month which was a pretty good deal at the time but just seems insane now. The other half of the duplex was occupied by this weird old sour hippie lady who always looked like she just came out of a wind storm and would shoot me dirty looks all the time because I liked to have my windows open to the sun and the air and I'd play music and sit on the porch. The house was right on the park that runs through the middle of the 78741, and I'd go run around the park for laughs. I'd ride my bike over to Barton Springs and swim there.

Mostly I'd go to the little pool right in my park. Austin has a great old city public pool system from back in the glory days of America when our government built beautiful structures for us all to share and to improve our cities. I would go up to the pool and just lounge around. Swimming a session just for exercise is awful, but having a dip every so often in the heat is divine. I would swim 20 laps or so, then lay out in the sun a bit, then swim some more, then lay out. I hadn't yet exhausted the Nabokov body of work, so I would read one of his books and exult in the joyous beauty of his sentences. The pool was amazingly free of kids (maybe I went at adult lap swim times?) and generally nicely empty, but there were usually a handful of soccer mom housewives who would oggle me in my speedo. I also did weird stuff back then, like I would swim my laps, then get out and do some pushups and situps on the side of the pool to try to make it a more balanced workout. I feel very conflicted about things like that. I wasn't doing it to be a spectacle, it's what I would do if noone was around (in fact if noone was around I would do even weirder stuff, like swim in circles underwater), but I am aware that it makes a spectacle which I don't like.

I used to bike up to Mount Bonnell. That's actually an incredibly short easy ride, I now know, but at the time I thought it was a decent workout because I was a terrible cyclist and had no concept how far behind a real biker I was (and my bike was really slow and heavy, I now know). I think that's also the summer that I passed out from heat stroke biking up a hill in west Austin. I think it was Redbud Trail, which is a decently hard hill, but it was over 100 and I wasn't being careful about keeping myself cool. I was biking along the side of the road and started seeing black dots in my vision. Then all of a sudden I woke up and I was lying in the shoulder and I don't know how I got there and I had an intense head ache (from dehydration not from impact). (I was to have a similar but much scarier incident many years later on Hwy 46 near Cambria, going down hill very fast - it's a long steep hill, about 3000 feet in elevation change, going probably 40 mph I was riding along and feeling really bonked and started dozing and did that thing where your head nods off and suddenly you wake up and didn't realize you'd been asleep).

That summer in Austin was the first time I started doing hard functional workouts. I went to "Big Steve's Gym" which was the most awesome gym I've ever seen (I hear it's been renovated or changed owner's and is no longer the same). It was this run down old power lifting gym. It had no cardio machines at all, just a ton of old dirty weights, chalk buckets everywhere and chalk spilled on everything, and there was a boxing ring on the first floor that noone ever used. I would go in and squat 100 and watch guys squat 700+. Big Steve used to be a bench press champion, and he was training up this midget while I was there. The midget was going for the world record in bench press ratio - that's the ratio of what you bench to your bodyweight. Competitive bench press is a weird thing because it's defined relative to your body geometry, not some abstract distance. You need to push from your chest to your arms being locked out. That means that if you can get a big enough chest and short enough arms, you barely need to move the weight at all. All top end benchers have huge rib cages and short arms, but this midget was the most extreme case I've ever seen. He was literally built like a barrel, he had a huge rib cage, like an Andre the Giant huge rib cage, and then short stubby legs and arms. He only had to press a few inches to do a legal bench. I think he was going for a 3.5 ratio or something, I don't know what came of it, I was definitely the weird kid outsider at that gym and all the big old powerlifters just tolerated me as a necessary evil to pay the rent.

I did weird workouts in the park. I watched GI Jane around that time and decided I would replicate some of that training, so I would do stuff like hang upside down from the monkeybars and do situps. I also did this funny thing for legs. I lived on a bit of a slope, maybe 5%. I would put my car in neutral and then get behind it and push it up the hill as far as I could get, then let it roll back down and do it again. I also read a book on plyometrics so I would do weird plyometric training stuff like clap pushups and fast high jumps and hammer throws in the park.

09-30-09 | The Olympic Peninsula Sucks

The Olympic Peninsula is a load of crap. It's touted as this wilderness wonderland, but it's really all pretty boring. Yeah the coast is nice, but if you've seen the Oregon coast or the Northern Cal coast or Big Sur it's pretty much all the same thing. Big rocks and waves crashing, yeah it's starkly beautiful and all, but it's such a kleeshay. Can't you do something more creative, nature? That said, the various beaches are beautiful in comparison with the incredible letdown that is the Hoh Rainforest. Ooo look it's a bunch of trees with slightly more moss than other places. And there's a mob of tourists and no good trails or views. It's ridiculous.

There are some outstanding sights on the OP, but they aren't the nature :

1. Just driving the 101 around. It's usually pretty low traffic and not many cops, so you can really let the car run free. The roads are swervy and hilly enough to be fun but not dangerous and the pavement is pretty solid. It becomes a real treat at twilight as the sun starts to go down. Because of the rolling hills and our northerly latitude (longitude? what-the-fuck-itude?) the sunset lasts for hours and makes the tree-covered hills like spiky sillhouettes against a field of color. The biggest risk is deer.

2. The Satsop "Development Park" (the cancelled nuclear power plant). There are the huge cooling towers that you can get into and they're just gorgeous; it's a magical place, the echo of sound on the inside is eerie. Oddly they are trying to convert the site into an office park for high tech companies (lol, good luck with that goofballs, it's in the middle of butt fuck nowhere) ; the sweet thing is they are leaving the cooling towers in the office park : About visiting it ; NY Times article on it ; Satsop Development Park Property Map

3. Some neat little poor country towns like Blyn, Discovery Bay, or La Push. I like the way these places look like time is not passing at all in them. There's a dog wandering around between houses (is it a stray?) , a boat upside down in the front lawn that could have been there a week or ten years.

4. Aberdeen. This town is just gorgeous if you like sad urban decay and the glories of yesteryear. It was obviously a thriving harbor town around the turn of the century, but now literally the entire downtown is boarded up. There are some great old iron bridges, stone buildings, and a palpable feeling of loss in the air. (Port Angeles has a bit of this too, but Aberdeen shines because it was actually nice before it fell). I keep just passing through and always swear that next time I'll spend more time here.

If you do have to go to the OP for some reason, go to those places and just ignore that national park. If you want to come to WA and see some natural beauty, I recommend North Cascades, perhaps off Mountain Loop Highway or up off Mt Baker Hwy.

Theoretically in season you should be able to get some good Dungeness crab somewhere, but I can't actually figure out what the season is or where to get good ones. What the fuck, that should be a major draw.

The trip was a great joy just for the little random moments together in odd situations, but it did a fucking number on my back. Sitting in my car is really bad for me, and the fucking shitty hotel beds hurt me really bad. It's a few days later now and I'm still suffering from knots and a pinched nerve that's really fucking me up. Oddly backpacking was far better on me than road tripping. Actually just taking my thermarest sleep pad and using that in a hotel room would be better than these awful hotel beds that sag in the middle and have wierd pressure points.

09-29-09 | Aliasing is Pretty

I made some images of aliasing/moiray patterns : (BTW inherent to these images is the fact that they don't look at all right except at full zoom since the image structure is created by the pixel grid sampling aliasing, so click through). You may be surprised to know what these are an image of.









Answer : these are pictures of sin( x^2 + y^2 ) , that is, sin( r^2 ) a very simple radial trig function. You don't see the function at all, the images are 100% aliasing as the radial sin is scaled to very high frequency.

09-28-09 | Clubs

We went out to a club for Decibel Fest. A few thoughts :

One is : my god, most DJ's fucking suck so bad. All you have to do is play some banging tracks and string them together smoothly and the crowd will be happy. That's just the basics. Anything above that is gravy, but instead people try to be fancy and unique and just fuck it all up with long silent pauses and crazy annoying screeching noises or long breaks where the beat disappers. WTF DJs get your shit together. I have zero experience but I'm sure I could do better with about 24 hours prep to get tracks together and just fucking play them on my laptop and do beat-matches crossfades. The headliners were the Wighnomy Brothers who I guess are pretty famous and reputed, but good god they sucked balls and it took a real force of internal positive energy for us to keep the good vibes cranking.

(of course you see the same thing in coding all the time; it's a classic blunder for people to assume that the obvious easy solution is not good enough. To accelerate a lookup, of course you should just use std::hash_map and spend your energy making sure you use it well and have a decent hash_function for it, instead people will right off the bat jump into making a custom structure which is much worse in many ways, slower and buggy. Almost all the time you should just take the simple obvious solution and focus on executing it well rather than trying to be really different).

The other is : my god Seattle fucking sucks so bad. The people here are fat and ugly, they don't dress up, everyone's wearing flannel and jeans and they're all beardy and sweaty and gross and worst of all, they're not dancing, they just fucking stand around and act all emo. God you all suck so bad. Fucking make an effort, go to the gym, dress up, and just have fun and dance. It's very comfortable here, it's easy, and I have a nice life and I like my friends here, so some times I get lulled to sleep and forget how lame it really is. Whenever I try to go out to the clubs it reminds me how much I miss SF. I miss the crazy Saturday night at the The Transfer (which I guess is gone) where the gays and the hipsters and all the wacky people from the neighborhood would come together in a tiny space and get all sweaty and dance like their life depended on it to awesome cutting edge indie electro that was smoothly mixed and always pumping.

I think a lot of people who try going out and don't like it have the wrong attitude about it. It's never going to be some amazing extravaganza that whisks you off your feet. It's always what you make of it. It's also not a big deal on any given night. If it sucks, just bail out, and who cares. It's way more fun if you just go to check things out, if you like it then kick it, if not, then leave and go somewhere else or try another night.

I also ran into another aggressive skeezer and wish I would have handled it differently. This is a very common occurance because clubs are infested with a virulant swarm of skeezers; it's happened several times with various different girlfriends in the past. The basic move of the skeezer is something like this - they come up to your girlfriend ostensibly to ask some question or make some "funny" comment, and they use the excuse of the club noise to stand way too close, and often even put an arm around her waist as they whisper in her ear. Often they will even hold her close so that her boob pushes into their body. They act all casual and small-talky, but they blatantly ignore the fact that you're standing right there and pull some dominance game bullshit. My usual lamer reaction is to give them the stink eye and subtly pull my girlfriend away from them after it's clear they're sticking way too close for way too long (they try to stick like velcro so it often takes a push to their chest to actually get them to separate). But it always takes me a minute or two to react because I spend the first minute in a state of shock thinking "WTF, really?". Anyway, I wish I'd just punched the guy straight in the nose. Or maybe not that, but perhaps grab him by the ear and pull him off her. Fucking skeezers.

I've also never in my life actually seen a girl step up and tell the guy to back off herself, like "hey easy with the hands bub", they always just laugh uncomfortably and take it because they don't want to turn it into a big deal. The skeezer exploits people's inertia and desire to be polite and unwillingness to escalate to a confrontation. Yeah, the skeezer's invasion of space and subtle groping is not a huge issue, so it's easy to just let it slide, but that's exactly his intention, that's how he gets you. It's like the guys who are subtle dicks to you, they drop half-joking put-downs, they try to slide right under the edge where it's not a big enough deal for you to say anything about it, and then they keep pushing that edge farther. Of course all the crooked corporations do the same thing, they charge you little fees here and there, they take away your rights and stick you with contract provisions, each one of them is too small for you to get too bothered about, so you just sit there and take it like a little bitch.

On a tangent, if you're a girl you should know the probability of a guy talking to you is directly proportional to his skeezitude. Obviously this creates a very bad sampling bias if you're trying to meet good human beings. It's pretty much a 100% gaurantee of skeeze if he touches you while talking to you. Most girls find it entertaining when the skeezer jokes with them and touches them, they like the fact that he's showing interest in them, making them feel desirable, they like the fact that he's energetic and seems to get the attention of many people in the room. I don't mean to imply that girls are unique in their failure to appreciate population filtering and sampling bias; I haven't met many guys who apply logic to probabilistic human selection in dating.

ADDENDUM : Seattle is the only place in the US I've ever seen people line up for drinks at a bar. It's really bizarre to me and I think emblematic of the repressed weirdness of this city. Like you have a long empty bar with a few bar tenders behind it. Normal behavior in 99% of the US is just to go up to the bar at any empty spot and make yourself visible to the bartender. The bartender sort of notices who got up to the bar first and will serve people first-come first-serve (or some approximation thereof).

In Seattle, people go up to the spot on the bar with the rubber pad and just stand there. If someone else wants a drink they stand behind that person. If a bunch want a drink, an actual line forms, like people waiting to order drinks at a coffee shop or something. It's incredibly bizarre. It freaks me out every time I see it.

I despise Toby Young but I thought it was pretty awesome that he was saying "pie ella" for paella.

WTF it's simply impossible to buy fish in Seattle. So far as I can tell there are only two places that carry even edible fish : Uwajimaya and Mutual Fish. Mutual Fish closes at 5:30 , Uwajimaya closes at 6:30 , neither of which I can make on my way home. (so far as I can tell the Uwajimaya fish counter actually closes at 7:00 but any time I've been there after 6:30 they're already shutting down and don't want to be bothered).

09-26-09 | Habits / Code Inlining

There's been an off and on discussion with some peers in email about the issue of putting small code functionality directly inline in a bigger function vs. splitting it out. I don't think it's actually an interesting topic so I don't want to repeat it here, but it made me think of something I want to write about.

Basically the contention is that even if you have a logically separate piece of work, sometimes it's better to go ahead and write it inline in the larger function, because it makes the code flow more linear and imperative and thus easier to read and debug. There are also obvious disadvantages to writing code inline - less strong separation of functional bits, temptation to copy-paste code, less ability to debug and test functions independently, etc. etc. All the pros and cons are obvious and thus not interesting.

The key point to me is the issue that while code inline may be a win sometimes - it's a big loss at other times, and it requires you to make a careful smart logical decision about which way to go.

That's very bad. I think most smart people as they get older come to the realization that you shouldn't put yourself in positions where you have to make careful logical decisions over and over. Instead you should craft habits and rules for yourself that help you do the right thing without having to make careful decisions.

One obvious case that most smart people figure out is diet and exercise, or booze or other indulgences. Yes, it's perfectly fine to have a some dessert once in a while, but if you open that door for yourself you're putting yourself in a situation where you are consciously making a decision "is it okay for me to have cake today?" and you will inevitably get lazy and make the wrong choice sometimes.

As usual the best analogy is poker, and it's how this point was really made real for me. Smart people often start out playing poker trying to logically reason out every single action and think they don't need to be constrained by simple rules or habits. That's great if you really are correctly thinking through every situation, but you inevitably get tired or lazy or make mistakes, and if you're not constraining yourself with rules, you can make huge mistakes.

For example, there might be cases where the best play is to limp aces up front, or to not reraise with aces, but correctly making that decision requires a lot of careful thought, and the upside to making that decision is pretty small, while the downside to doing it in the wrong situation is very big. It's best to just make a rule for yourself that you always raise or reraise. It simplifies things, it removes decision points and lets you focus on more important issues. It might be +EV to call raises sometimes with hands like 68o, but it's best to just give yourself a rule that you never do that.

To be clear - these rules are specifically non-optimal. By making rules for yourself you are intentionally choosing to not try to be 100% optimal, so of course someone can say "you could have done something better there". That's not the point. The point is that if you try to make perfect decisions all the time you will occasionally fail very badly.

Winning poker play (or good coding, or good life living) are largely about making it easy for yourself to do the right thing.

09-25-09 | Motion Search

I just read the paper on Patch Match and it makes me angry so I figure I'll write about the motion search method I'm developing for possible future use in the new RAD video stuff. PatchMatch is just so incredibly trivial and obvious, it's one of those things that never should have been a paper and never should have been accepted in a journal. It's a great thing for someone to write on their blog because you can describe it in about one sentence, and most experts in the field already know the idea and are probably doing it already. (I will say the good thing about the paper is they do a good job of gathering references to other papers that are related, such as stuff in texture synth and hole filling and so on which I find interesting).

Here's the one sentence version of PatchMatch : Seed your match field with some random guess or shitty initial matches; improve by incrementally propagating match offsets to neighbors and trying small random deltas to find improvements. (it's an absolute classic spin network magnetic moment relaxation kind of problem).

Here's what I've been doing : start with a match field set to all nulls (no match found yet). Then incrementally fill it in with matches and propagate them to neighbors. It proceeds in a few steps like this :

Step 1. Use computer vision methods to find feature points in a frame. Match feature points to the previous frame. This bit is a bit tricky and tweaky, you only want to make matches that you're pretty confident in. Note that this matching is done based on a "characteristic" of the feature point which has no distance limit, and is also somewhat immune to rotation and scaling and such. Sometimes this step finds some very good correspondences between the frames, but it's sparse - it only has high confidence at a few places in the frame, so you can't use it to find all the block matches (and you wouldn't want to even if you could). Generally this finds around 100 vectors.

Step 2. Find "distinctive" spots in the frame. The goal is to find some spots that are not degenerate - eg. not flat patches, not straight edges. The idea is that these are places where we can likely find a good motion vector with high confidence, unlike degenerate areas where there are lots of equally good match vectors. I use two mechanisms to find distinctive spots : one is the computer vision feature points that were not already used in the first matching step. The second is to take the "cornerness" map of the image using a Harris or Hessian operator on the derivative of gaussians (this is a lot like an edge map, but it kills straight edges). Find the top 5% highest cornerness values that are local maxima, and use those as distinctive spots. All of the distinctive spots do a long distance brute force block match (something like radius = 16 or 32) to try to find a good motion vector for them.

Step 3 : Flood fill to fill in the gaps. We now have presumably good motion vectors at a few key points in the frame. Go to their neighbors and search for match vectors that are close to the neighboring one that we already found. Put that in the blank and push its neighbors to the queue to continue the flood fill.

Step 4 : Relaxation pass. (this is not critical). We now have a motion vector everywhere in the frame. For each match vector in the frame, look at its 4 neighbors. Examine match vectors that are near my 4 neighboring vectors. If one is better, replace self. Continue to next. Theoretically you should do this pass a few times, but I find 1 or 2 is very close to infinite.

The key thing is that motion is usually semi-coherent (but not fully coherent, because we are not really trying to find true motion here, but rather just the best matching block, which is a lot more random than true motion is). By finding very good motion vectors in seed spots where we have high confidence, we can propagate that good information out to places where we don't have as much confidence. This lets us avoid doing large brute-force searches.

BTW I really do not understand the point of all the "diamond search" type shit in the video compression literature. It seems to just find really shitty motion vectors and is not making good use of the possibilities in the bit stream. Especially with GPU video encoding in this modern age, doing plain old big chunks of brute force motion search is preferrable. (yes, I know it's for speed, but it's a poor way to optimize, and the high quality encoders are still non-realtime anyway, so if you're not realtime you may as well take some more time and do better; plus the vast majority of use of non-realtime video encoders is in an encode-once decode-many type of scenario which means you should spend a lot of cpu and encode as well as possible).

With this method I find motion vectors using local searches of radius 8-16 that are the same quality as brute force searches of radius 50-100, which makes it about two orders of magnitude faster (and higher quality, since nobody does brute force searches that far).

ADDENDUM : To give this post a bit more weight, here are some numbers on quality from my video coder vs. brute force search radius :

 -s16  : rmse : 9.3725 , psnr : 28.7277
 -s26  : rmse : 9.2404 , psnr : 28.8510
 -s48  : rmse : 9.0279 , psnr : 29.0531
 -s64  : rmse : 8.9171 , psnr : 29.1603
 -s100 : rmse : 8.7842 , psnr : 29.2907
 -s9999: rmse : 8.5294 , psnr : 29.5465

(-s16 means it's searching a 33x33 grid for motion vectors) (-s9999 means it searches full frame).

The above described iterative feature point propagation method gets

 -sfast: rmse : 8.8154 , psnr : 29.2600

BTW for doing full-frame brute force search you obviously should use a block-space acceleration structure for high dimensional nearest neighbor search, like a kd-tree, a bd-tree (box decomposition) or vp-tree (vantage point). High dimensional spaces are nasty though; the typical idea of "find a cell then walk to its immediate neighbors" is not fast in high D because you have O(D) neighbors.

09-24-09 | Indian Summer

We're having a little indian summer in Seattle. I keep thinking it's the last nice day before the drear kicks in, but we just keep getting more hot sunny days. I really wish I could get out and get a good bike ride in while the weather is still good, but I don't feel like I can take the hour or two off work, just fucking being alive and doing the minimum of errands takes so much damn time.

A few days ago when we had a nice hot night, I went late night skinny dipping in lake wash. (lake wash in the P.N.dub bitches). It's long been a dream of mine, and it finally just happened. All my fantasies are become reality these days, effortlessly, easily, they just happen.

My fucking phone continues to randomly decide not to send texts (they just go into pending state and then never retry). It keeps leading to bad social misunderstandings because someone will send me a text, I'll reply, but it doesn't actually go out, so they think I'm just ignoring them, and then I think they're ignoring me.

My continued boycott of Facebook and Apple is becoming more ridiculous by the minute. Aside from facebook being specifically designed to suck your time, my biggest complaint about it and Myspace is the fact that they are not open and try to trick you into registering by making them visible to members only. I try to boycott all web sites that are members only, so GORP, trails.com, etc etc shit that won't show you their content without registration can fucking suck my cock. There are bands that only list their tour dates on their facebook site now, restaurants that put their current menu on facebook, etc. Facebook is not part of the open internet! All of you need to get some balls and join the boycott with me.

I'd like to be able to set up Google to never show me results in protected sites like Experts Exchange (fuckers) or the IEEE (fuckers) or Facebook (fuckers). There should be a "free content only" option. We should have a "free the internet" movement.

I am a big believer in not trying to understand things that are too complex to be understood or that you don't have enough time to really dig into or that you don't have all the information about. If you try to reason about something without a full understanding, it's easy to make very big mistakes. In those cases it's much better just to back up and take an empirical holistic view.

(the exception of course is researchers who are trying to expand our area of understanding, but they should also not be taking cutting edge research and telling the public about it or making conclusions from it; basically my contention is that research into complex topics like the body should have a waiting period where the scientists can hash it out before we try to draw conclusions from it)

For example, I know sugar and fat are bad for me (in large quantities). My body feels bad when I eat it. I don't think we have a solid understanding of all the metabolic pathways involved in digestion and how the body's internal regulatory systems respond to various inputs and how that all interacts with your exercise schedule, and I think that paying too much attention to the limited set of details that you might know can only lead to skewed pictures and leaps of logic that don't match the simple big picture. I really don't believe in paying much attention to all the nonsense about lycopenes or omega-3's or alpha-stupidol or whatever. Maybe some of that shit would be good for you, maybe not, the science is incredibly weak. It's funny to me the exact same people who will make fun of the now-disproven food fads such as low-sodium and low-cholesterol are the same people who try to eat almonds and blueberries and believe that canard about red wine being good for you because it appears to benefit something in some specific pathway, without understanding how it affects feedback systems long term.

For another example, when the Fed was holding interest rates so low for a long time when we weren't in recession and everybody was convinced they could get rich quick by buying a house, you should be able to see something is going horribly wrong. You may not understand the exact mechanisms of the financial system that will lead to problems (in fact, maybe nobody understands all the factors at play and self-balancing mechanisms and positive and negative feedback loops in the financial system any more), but focusing too much on logical details and making risk analyses and macroeconomic models can easily lead you to false conclusions. I think you see the truth better by just ignoring all that rigor and detail and stepping back to the big picture and saying "something is obviously wrong here and it doesn't matter if whether or not I can say exactly what or how it will blow up".

Another great example is politics and world affairs; it's tempting for the smart rationalists to think "hey if we topple this regime or back these rebels or sabotage this political campaign then these other things will logically follow" , but it's just too complex a system and they fail to account for various factors, and through the blidness of logical reasoning without a full model they wind up making mistakes that are very large.

It's another thing that was really hammered home for me by poker. Yes if you really really understand what's going on, then you can reason from logical first principles and come to new conclusions (BTW you probably don't really understand what's going on, so don't do this). Failing that, you are best off using very simple empirical concepts aka "common wisdom" and "rules of thumb" and such. If you try to figure out how to play using logic and math without 100% understanding, you will be much worse off than if you just followed the wives tales.

Commuting fucking blows so bad. I've been waking up really happy recently, full of energy and peace. Especially after a night spent with N I wake up in a state of bliss with just a content glow. I have coffee, read the paper, eat eggs, and I'm in love with the world and the fresh air and the sunshine. And then it's all wiped away and replaced with weariness and anger.

I've been doing really shitty work for RAD recently. I go in and try to make myself focus, but my mind just wanders, I write a little code and just lose track of where I am. I'm sure a lot of it is just because I've been absolutely exhausted every day for the past month because of backpacking or camping or just because I'm staying up super late with N every night painting the town red. But it makes me wonder if my days as a productive coder are done. And even if they're not done now, maybe they will be soon.

I definitely find that the happier I am in my outside life, the more I take care of my body, the more I socialize and have hobbies and exercise - the less I want to work and the less I can focus on work. I'm most productive when my outside life just blows and I'm lonely and I neglect my body and just sit at the computer all the time.

There's this myth that happier more balanced coders are more productive. Bullshit. The family man with a social life who exercises is maybe putting in 4 solid hours. The nerd who slams caffeine and never leaves his desk and turns into a throbbing brain with an atrophied body (like a third stage guild navigator) is way way way more productive. Yes, the nerd might wreck his body or burn out and not be able to work at all any more, but that's not my concern if I'm an employer - at that point I can just fire him and get a new nerd. There's this myth that exercise gives you energy; yes, it does, sort of, but it's energy that makes me want to run around outside, not energy that makes me want to sit inside and write code. It doesn't even have to do with time necessarily, I can actually feel that when my brain is happy, when it's bursting with contentment and excitement like it has been recently, it can't think clearly and it's not interested in diving into deep contemplation.

I never want to manage people, I tried it, I can't do it. Some day if I can't code or manage, I pretty much have to leave software. But I have no idea what else I could ever do with myself. In some ways this industry feels like a dead end.

09-21-09 | Backpacking

Went backpacking over the weekend in the Cascades (Pear Lake to be exact). It was my first time ever backpacking and it went pretty well. I've never done it because 1. I didn't have the gear and didn't want to spend the $1k+ to buy all new gear, and 2. I was worried it would be bad with my fucked up back. It turns out I didn't need to worry about #2. A properly fitted backpack puts all the weight on your hips; my back felt perfectly fine the whole time. Also the modern sleep pads (Thermarest) are really cushy so there's no back pain from bad sleeping conditions.

After we set up camp, we scrambled straight up this face which was great dangerous fun :

And by pure luck we were at the right time for the masses of blueberries (*) everywhere, which were just becoming ripe :

They were all over in the understory; I guess I've seen the bushes before earlier in the year on hikes and had no idea what a bounty they would bear in the fall. Also the leaves turn bright red, which creates great patches of color contrast against all the evergreen trees.

(*) All the locals here seem to call them blueberries or "mountain blueberries", but I'm not entirely convinced that's the right appelation. Certainly they are a berry in genus vaccinium, but it seems very hard to tell whether a given vaccinium should be called "blueberry" or "huckleberry", the distinction appears to be pretty random. In any case, these things were vastly different from the commercial cultivated blueberry, which I find to be mealy and insipid. These were just bursting with flavor, with a twang like a sour apple candy or tropical fruit flavor notes.

Another random tidbit as a note to myself : the first night in happened to be a moonless night, so the stars were absolutely spectacular. That was unintentional and just luck, but it's so cool it's worth planning and doing on purpose.

09-17-09 | Life

I returned a bunch of stuff to REI that didn't work well on our last camping trip. It's so great that you can use stuff out in the field and still return it for a 100% refund. Of course I have spent like $2000 at REI recently in a big fucking yuppie materialist douchebag capitalist ejaculation of money, so it's not like they're really doing me any favors. If you search for "REI refund" on Google you'll find site after site where people brag about how they abuse the policy. You fucking cock holes. First of all, it's a really nice thing REI is doing, they're showing trust in the customer, and you're just proving them wrong, taking advantage of someone being nice and trusting is such a fucking awful thing to do. Second of all, you're just stealing from all the other customers. You're not stealing from REI, you are literally stealing from the people who are honest and respectful of the policy. So many people think that taking advantage of a collective is something to be proud of. My god, you're not fucking clever or bold, we can all see how easy it is to get free rental stuff from REI, you aren't the first genius to come up with that. It's not something to be proud of. People love to brag about cheating on their taxes, or getting "free" stuff from health care, or how they avoided paying some tickets or fines. I remember in California lots of people at Oddworld would brag about they falsely claimed earthquake damage on their homes to get money from the government. You fucking tool, you're stealing from me. People are so disgusting, I hate you all so very much.

I wrote a while ago about the ridiculous speeding ticket I got for going 65 on the god damn freeway. Well, the lawyer that makes tickets disappear made it disappear. That's nice and all, and certainly that ticket was bogus and deserved to go away, but I feel bad about the whole situation. For one thing, the ticket disappearing had nothing at all to do with the fact that it was actually a ridiculous ticket. She just makes any ticket disappear regardless of merit; that bothers me; many speeding tickets are well deserved and people should have to pay. Instead the people with enough knowledge and money to hire the lawyer get no tickets (aka the rich) while the young and the poor get fucked. For another thing, I have been driving like an absolute lunatic recently, I've done 100 right past cops a few times and have made some very sketchy passes, and somehow I have avoided tickets in all those cases. I feel like I deserve a ticket, and even though that one was bogus it was my ticket for all the times I didn't get caught.

I think people who wear workout or hiking clothes around town are super ridiculous loser douchebags (particularly guys who wear compression shirts around town, though wearing hiking gear is pretty bad too; even the girls in yoga pants are pretty retarded). However, I wore some hiking breathable underlayer thing around yesterday and it felt pretty damn good. I was warm and yet not sweaty. I guess the thing I hate is that a lot of people wear that shit because they're trying to show off how fit or outdoorsy they are, oh look at me in my fucking hiking pants I'm such a good Seattleite, but it actually is pretty damn comfy and functional.

N is super idealistic. It reminds me of how I used to be. She doesn't want to use her name or sex or influence to get ahead. She wants communities and respect for all and bikes and pedestrians and help for the poor. She's anti-materialist and anti-capitalist; she believes in making do and doing things yourself. I've given up so much of that, I used to be such an idealist when I was young, I boycotted companies that I thought were evil, I refused to pimp myself, but I gave it up gradually bit by bit in the name of practicality. I used to rail against the system in hopeless self-defeating utopian ways; over time I gave into realism and realpolitik and reality. I love feeling like a dreamer again.

When I lived in SF during unemployed time, I wanted to intentionally live like a broke person (I actually was pretty broke, but wanted to try to live as cheaply as possible for the experience of it). We found furniture and carpets on the street, we shopped at thrift stores, ate at divey ethnic places, for fun we hung out in the park or took a walk. It was glorious. My girl was not really appreciative of the beauty of living like you're broke. I find it incredibly romantic. I have this fantasy vision in my head of being a couple of poor immigrants in the big city, struggling to get by, working horrible jobs, living in a rotten apartment, being cold and dirty, but getting through it all together, helping each other, and being happy because we have each other. She would rub out the horrible knots in my back that I get from working in the factory all day, I would rub oil in her hands that are dry from washing dishes all day. We would chase each other around the apartment and wrestle and tickle and fall in a heap and have sex and life would be grand.

Before I was 18 or so, I had magic powers with children. They would flock to me; I would go to a playground to play myself, and children would come up to me and talk to me and play with me. At family gatherings I would always prefer to sit at the kid's table and hang out with them the whole time. I couldn't really connect to people my own age or adults, but I felt natural with kids; they were simple and open and honest and fun. I was great with babies and was sure that I wanted kids of my own some day. At some point, I lost that. I imagine a cheezy movie where all the kids are in color and the adults are black and white, and up to 16 or so I was in color, but then it started fading until at some point I went almost coal black. N is a buzzing conflagration of rainbows; just being near her, I'm drawing color back into me.

Part of me getting my vibrance back is rejecting situations that beat me down. Over these many years I've let my spirit die by caving into tedious boring normalcy. Girlfriends and friends and everyone want me to do all these things that are just so awful, and I'm supposed to have a good attitude about it and find the fun in it and all that, and I could to some extent. I found ways to turn off my real opinions and just be open and appreciate the upside. That has been good for me in my personal growth, but it also just makes you constantly half alive. I feel much better when I can be wildly passionate, either madly love something or hate it; this "meh this sucks but everything sucks and I can tolerate it" is no way to live.

N loves to bike, perhaps even more so than me; she has a pure simple love of the feeling of moving freely in the air under your own body power. We roam the city like it's our own personal playground. Travelling together by bike has a wonderful equality about it; it's not like a car with a driver and a passenger, you are both under your own power and can take turns leading or separate briefly and come back together. Biking makes it so much easier to stop and start all the time and experience the world and explore. I feel like we're birds, flitting around, swerving, separating, chasing each other, flying along side each other, one racing off, the other catching up. It makes everything so much more fun, because instead of just going to X, we're going to X by bike. Sometimes we never reach the destination because we see something else along the way and wind up liking it better; it completely changes the feeling of every excursion from stressful type-A goal oriented to a fun open-ended ramble.

I like to yell at other cars when I drive. It really relaxes me, relieves the stress. Rather than chasing them down and cutting them off and trying to run them off the road, I just have a good yell and then I feel okay. When someone else is in the car, though, I can't do it, because it freaks them out, makes them all stressed. They take it the wrong way - they think the yelling is making me angier. Quite the opposite, *not* yelling makes me much angier, I have to bottle it up inside, and the small bits of rage pile up into a festering parasite that consumes my guts from the inside. The only other alternative is just to chill out and turn off my brain and not care, but that's just so soul crushing. It's like being some valium/prozac zombied out drone. "Oh, I was trying to turn left and pulled out in the intersection, but I couldn't go because some nice person decided to run the yellow going straight and prevent me from getting my chance to turn left across a busy street. la di da. that doesn't make me angry at all." No thank you. When I'm alone I can do lots of weird stuff to let out my rages. If some social misunderstanding bothers me, I might just go outside and do some sprints and pushups. If I'm with someone, now they're thinking "oh what a weirdo he went outside while we're in here talking" and it's now become way bigger of a deal, while if I was alone it would diffuse the problem for me. I constantly don't do what I want to or need to because the people around me can't understand it and respond to it right because they're closed minded and conformist. No more.

09-18-09 | Ross Lake

Last week we went to Ross Lake in the North Cascades. It's a pretty incredible experience. You have to catch a ferry at Diablo Lake (BTW the dam and the water at Diablo are worth seeing even if you do nothing else; Diablo Dam is right off Hwy 20 so if you're driving the 20 for some reason, pull off and stop right in the middle of the top of the dam (the access road goes right across the top of the dam); the water is glacier fed and is this cool milky turqoise). Then you get taken by a truck up to Ross Lake above Ross Dam, then you get picked up by a water taxi from the Resort. They gave us kayaks and took us out to a camp site on the lake. The lake is huge and there's no boat ramp on the US side, which means everybody has to portage in boats or take the ferries like we did, which means it's very empty (not full of rednecks in speed boats like most lakes in the US). We were often all alone on the lake, or with just a few other kayaks or quiet fishermen around.

The result of all the effort to get in is you get a campsite way out all alone in gorgeous country, as good as anything you could backpack to, but you can take more gear and you don't have to carry it, because it's all ferried by boat, and you get your kayaks when you're out there. We had amazing adventures and a magical fantastic time out there; I won't tell you about it. It's definitely worth doing. Weekends in the summer are supposed to be pretty bad. You should go midweek, or the resort guys said the best times are actually June or October; the weather is not so great but it's very empty. In June there will still be snow on the craggy peaks all around and waterfalls of snow melt running off them. The best camp sites are May Creek or Roland Point where there are only 1 or 2 camp sites all by themselves. All camp sites are first come first served at the ranger station backcountry permit signup place; that's why you really need to go on an off day or you'll be stuck in the awful big group camps that completely spoil the point of the arduous journey (Big Bear camp looked pretty awful when we kayaked past it, for example).

N on Ross Lake :


Some external photos :

Diablo Gorge
Ross Dam
Ross Dam 2
Skagit Gorge power station

I've always wanted to have a 4x4 all pre-loaded with camping gear in the back, so I could just hop in an go anywhere. The worst thing about camping is having to get all the gear together and load up the car. The second worst thing is having to go to fucking awful car camping sites. I'd love to just be able to drive up mountain roads and stop somewhere in the woods.

A few links :

Tom's View From Ross Lake Resort How to get here
Ross Lake Resort Frequently Asked Questions Rockport, Washington
Ross Lake Recreational Area - Upper Cascades Dams
Ross Lake Kayaking with Anew Outdoors
Ross Lake - WA - paddling trip report
Paddling Ross Lake - WA - paddling trip report
Pacific Northwest Seasons Ross Lake Paddling in the Path of Beat Poets
NWHikers.net - View topic - Little Beaver-Whatcom Pass-Copper Ridge 813-17
NWHikers.net - View topic - Desolation Peak 9-11-04 (lengthy... with pics)
NWHikers.net - View topic - Canoeing on Ross Lake (and maybe some hiking)
NWHikers.net - s090819Lodgepole-Camp
North Cascade Mountains, Washington State, USA
Fortress of solitude The arduous journey to Jack Kerouac's onetime retreat
diablo love - a set on Flickr
big beaver zen - a set on Flickr
Bellingham Whitewater » Exploring Ross Lake Part 1 Lightning Creek
Backcountry Permit System Explained
Adventure Kayaking Inland Waters of ... - Google Books
A long weekend on Ross Lake Chattermarks
A little slice of heaven at Diablo Lake
5 days on Ross Lake - Alex Wetmore is always busy with something...

09-11-09 | Quit wrecking things

Levi's redesigned the 501 around 2003 and called it "The Original 501" and ruined it. (not only did they switch to much cheaper materials and worse stitching, but they also ruined the cut and fit of it, it's all bunchy and saggy now and fits bigger). (and god damn you and your fucking abuse of a product id number - if it's a different product, change the damn number). (ADDENDUM : if anyone knows where I can find a model-year 2000 501 in 34x34 let me know; yes yes I've tried ebay but as usual the damn retarded sellers don't label their products clearly so I can't tell what models they're selling, they just write "used 501" which is fucking useless - *which* 501 ?).

Some car-designer jackass decided a key in the ignition was too fucking simple and decided we should have to push a button too, and that turn signals you actually deflect and lock into place was way too obvious and clear and instead they should just be deflection buttons which toggle some invisible state variables.

And now Melitta has ruined the most beautiful and simple coffee making device in the world :

The Melitta single cupper dripper is an icon of product design. It's simple, elegant, straightforward, and works marvelously. I make better coffee with a Melitta dripper than anyone else in the world has ever made. It's a perfect product. There's absolutely no reason to change it.

So of course, they had to redesign it. We now get this monstrosity :

This thing is such fucking balls. For one thing, it's an eyesore. It has crazy swooping lines that don't flow and it's all off balance. It hurts me just to look at it. Functionally, it is worse in every way. The base of the cone is suspended up higher above the brim of the cup, which makes it very top heavy and easier to tip (tipsiness was the only flaw of the original product, so good job on making that even worse). It's got these stupid windows that are to let you see into the cup I guess, but their actual function is to let steam out so your coffee cools as you drip it. And it's got no fucking handle, you have to grab it by the dumb curved lip, which puts your fingers right next to the steaming hot coffee.

You fucking morons. Stop taking your products that are perfect and making them fucking suck. It might be my biggest pet peeve of all when people take something good and intentionally and willfully make it suck (like when people take delicious sweet potatos that would be marvelous if you just did absolutely nothing at all to them and tossed them in the oven, but instead they boil them into tasteless mush and then coat them with marshmellows and bullshit and turn them into disgusting insipid candy filth).

I'm tempted to turn into this weirdo hermit pack-rat. When I find a product that hasn't been ruined by the fucking feature adders or the redesigners, I want to just buy up a lifetime supply and put it in a storage locker.

In positive product review news :

I went camping the last fews days and wore my new "smart wool" shirts and sweatshirt. It rained quite a bit and I got soaked, and the wool did exactly what it's supposed to - dried easily and stayed warm when wet. It's all natural, it feels nice, it's not all crinkly and plastic and bullshit like most modern outdoor gear, it's moss and dirt and animals and sex. I haven't gone in for the wool underwear yet like the Rivendell loons , but given the great success of the wool shirts I just might.

In other "god fucking damnit" news, the dickwad morons who lived in this house before me were subscribed to like every catalog known to man. Of course the post office won't forward or even stop delivering catalogs that aren't addressed to me (I talked to my mail man personally too). Of course the dickwad morons won't change their addresses with all these catalogs. So I now get 2-3 catalogs every day from fucking Mail Order Teddy Bear Express and fucking Two Useless Gadgets Combined Into One Superstore. Anybody who ever gets any catalog is a fucking dick tree-killer moron time-waster with bad life skills and priorities. Hello, have you heard of the fucking internet !? Stop getting catalogs.

09-08-09 | DXTC Addendum

Ryg pointed out that there are a few very important little details that I took for granted and didn't mention in my original DXTC postings , or was just not clear about :

One is that when I try all ways of hitting two given endpoints, I try both 4-color and 3-color versions. That is, given two endpoint colors C0 and C1, I quantize them to 16 bits, then try the DXT1 palette that you get from {C0,C1} and also the one from {C1,C0} (order of DXT1 determines whether it is 3-color or 4-color).

The second and related crucial thing, is that in 3-color mode, the extra color is transparent black. If the texture has no alpha at all, I assume the user will not be using it as an alpha source, so I treat the transparent black as just black. That is, I do color palette selection with alpha just ignored.

Apparently this is pretty important. I suspect this especially helps with the "4 means" method; if a bunch of the colors are near black, you want them to be classed together and then just ignored for the endpoint selection, so that they will go to the hard-coded black in 3-color mode and your interp end points will be chosen from the remaining colors.

09-05-09 | Brownstripe Fail

DSLReports speed test : 68 kbps down, 657 ms latency.

I think I actually might be better off with a cell network internet card for my laptop. It would be nice if it was easier to multiplex bandwidth. Then I could just keep the cable, get DSL, get a cell card, and send my traffic over all of them. I might actually have decent speed and reliability that way; having two connections seems like a nice protection in general.

I had a dream last night that I killed a man and decided to flee the country rather than face prison. It's actually a pretty common dream of mine; I usually dream that some guy hurts my girl and I beat him up and get carried away and wind up sticking my thumbs through his eyeballs. I'm not quite sure if it's a nightmare or a fantasy; I feel pretty horrified during the killing part but then kind of exhilerated about having to go on the lam .

Anyway, last night for the first time it occurred to me that it might not be so easy any more because of all the improved tracking post 9/11. It used to be trivial to drive across the border to Mexico, and then you could easily disappear into Central America and never return, but now they have the enhanced driver's license requirements - I wonder if they actually have the computer systems coordinated so that they can scan the bar codes at the border and see that you're out on bail and not let you cross ?

The other big issue is getting your money out of the banks. Again before 9/11 that used to be pretty trivial because law enforcement had no easy direct connection to banks, they would only freeze assets in extreme high profile cases. Now banks are doing way more reporting of large transactions. Most people know that banks report any transaction over $10k. That's actually been revised down to $3k, and the feds are pushing banks harder to examine and report suspicious transactions. I know this has given poker players some annoyance.

Even if you can get your money out, you don't really want to be walking around Central America with several hundred thousand dollars in cash. Again if you're on the run on bail, presumably American Express would stop cashing your traveller's cheques, though I doubt they'd actually get to that in practice. If you have a lot of money, you should of course keep a few gold bars in a safe deposit box in Antigua or some place. I guess a simpler compromise would be to keep one in your house and cross the border with it, but then you still have to deal with selling it and then having a mess of cash.

It seems to me that if you were a perpetrator of massive fraud like a Bernie Madoff or a Ken Lay or whatever, one of the first things you should take care of is getting yourself set up with some fake identities , a mix of various types of cash, some accounts in the fake person's name, so that when you finally get caught you can easily slip away.

I hate the idea of not being able to slip away and start over and change your identity. Not that I actually think I ever will, but feeling like you couldn't if you needed to is horrible.

09-04-09 | Bodies and Work

I've been super unproductive recently, and it's destroying my body. I've taken a lot of random bits of time off for dealing with various moving things (like meeting the fucking Brownstripe cable guy in the 8 AM - 5 PM appointment window) and even when I'm at work I feel like I'm not getting anything done. When I'm being nice and productive, I feel fine about taking time out to stretch and do PT and walk around and take care of myself, but when I'm not getting anything done, I feel like I need to force myself to keep sitting at the computer until I type some damn code.

Of course this is the exact opposite of what you should do to maximize utility - when you're in a good productive mode you should make the most of it and stay at the computer, and when you're not getting any work done anyway, you may as well get up from the machine and take care of your body.

I guess part of the problem is that I'm still stuck in the idea of the corporate "work day". I never used to think that way, but it was ground into me by overbearing producers in my past life in video games. The producers want you to get a certain amount done each day. If you're going slow, that means you have to work later, and conversely when you're productive and get everything done quick you can just leave (as long as the producers don't see you). Because they force you to stay sometimes when you don't want to, you have to compensate for that by leaving early whenever you can. Of course this is a horrible attitude, but it's beaten into you by the system. RAD has no real structured work schedule, and I believe it's much more natural for coders to work in a very spurty way. Coding at a high level is somewhat like an artistic endeavor, and many of us work best by spitting out a whole mess of work in a big flurry, and then taking perhaps a month with hardly any output at all. Of course in coding, nobody would believe that what you did in those two weeks of productive spurt was enough to justify taking it easy for a month, even if it was.

I'm extremely distracted by N. I can't stop thinking about her all day long at work, sudden memories pop into my head of our last time together, and fantasies and ideas and excitement about what we'll do next.

I went swimming today for the first time in quite a while. Every time I swim I think afterward : my god, that helps my back so much, I should do that more often. And then I don't. For one thing it takes so damn long because the pools are far away and you have to change and shower and everything. But it's also just really unpleasant, I hate chlorine and sharing lap lanes. If I lived somewhere with warm, clear, fresh water to swim in, I think I would be very happy. My body would be lithe and pain free, and a happy body is about all it takes. I sometimes have visions of me living somewhere hot and sunny with lots of fresh water streams and waterfalls to jump around in. I would just wear a speedo all the time and grow my hair long and get all tan and wear a shark tooth necklace like Manny Puig or something.

Fucking being on VS2005 / Visual Assist X has really slowed me down. Everything is so much harder, it fucking sucks balls at autocomplete. I feel like I'm walking through water. My mental flow is broken up all the time because I can't remember how to type a certain function name or keyword exactly and I have to go look for it. Back with VA.NET I could just jam, you go flow flow flow and the thoughts turn into code like magic. Now it's type, stop, type, stop, so awkward and jarring. It's hard for me to ever get any momentum. In fact I alt-tabbed over here to write this because once again I was in the middle of trying to write some line of code and fucking VAX failed to autocomplete for me and I had to go searching around to get the spelling right and it completely ruined my train of thought.

There was a point a few years ago, on Stranger at Oddworld, and also before that on Galaxy at home, when I felt like I basically had magical powers on the computer. I felt like code flowed perfectly and effortlessly from my finger tips, that I was some new kind of hybrid human whose brain had grown into the same patterns as C++; Homo Programicus. I could write good code faster than anyone in history ever had, and it came out bug free and fast and elegant the first time, everything was smooth and easy, thoughts became function. People around me were always complaining about the language or the compiler or whatever and it just perplexed me; I could never see their problems because code sprung forth from me like diarhea from tourists in India. That time is gone.

Feeling like your dreams will never come true crushes your soul. Feeling like you do the same thing every day crushes your soul. Feeling like you are putting great effort into something and it's not changing crushes your soul. Turning off your mind and shutting down your emotional response crushes your soul. Doing things that don't excite you and just faking it crushes your soul. Not saying what you think because the people you're with can't handle it crushes your soul. Having a routine of any kind, even if it's pleasant, makes you boring and bored. Doing only things that you're comfortable with and never feeling out of place or awkward or challenged makes you die inside. Without change something inside us sleeps and seldom awakens. N makes me excited to be alive. N makes me want to explore the world and see new things. N makes me feel like I can be myself and not be made to feel like a freak or have to apologize. N makes me feel a little bit out of control. When I'm with N I feel like anything is possible, we can go anywhere, do anything, and we seemingly do; I never know what's going to happen, and somehow it's me taking us off on adventures.

One of the wonderful exciting things about a new relationship is the uncertainty of each moment. For one thing just not knowing each other that well and learning new things, you keep getting new surprises. And each date there's a silent question in the air, not knowing where the night will go. On the first date that's the wonder in the air of whether you will kiss. Then you move onto the uncertainty of whether you will have sex. Then even after the first time you have sex it stays as a tense question on each date for a while, when you go out to dinner, or stay in for a movie, it's that much more exciting because you wonder "did she really mean come over and watch a movie or did she mean come over and watch a movie and fuck". Some people try to eliminate the unspoken tense question as quickly as possible by just talking about it or making it happen quickly. Those people are fools. (I've done that myself in the past, even in extreme ways; for example on first dates I've done the super early preemptive kiss to just get the tension out of the way from the end of the night). The mystery and tension is very enjoyable and should be savored, not rushed.

As the relationship progresses, that mystery and tension and uncertainty goes away, which is a damn shame. Girls who are really flighty and fickle are sort of exciting because they artificially create some of that tension - every time you take them out, even after dating for years, you don't know if they're going to get into a crazy mood and yell at our, or if they'll be all sweet and gropey. But that's just too annoying and immature to really be a substitute. Another idea is to just keep upping the ante. In order to have that tense silent energy, you have to both know that a question is in the air, but not talk about it directly. The question of "will she have sex with me" might be answered, but you can move onto another question like "will she let me stick it in her pooper" or "will she let me double team her with my friend" etc.

Of course the mystery isn't just about sex, it's not knowing the person completely so you don't know what new thing you might learn about them, what they might say about something; you're excited to find out all these new things. Most people are incredibly simple and static, so once you get to know them, you know what they think about everything and there's no more excitement or mystery to talk to them. I think of the old couples that literally tell each other what they think about things ( "you like this commercial" ). The solution to this is easier, you just have to keep learning and changing, getting new opinions and worldviews and experiences. Stay young.

09-01-09 | Hukkle

Hukkle was a pleasant surprise; it's very peaceful and gentle. It of course is in the vein of "Into great silence" and "The tree of wooden clogs" (both superb and much better), being movies with basically zero dialog that try to depict the feeling of life in a certain time and place. It's not actually about a "hiccup that travels through the village" as the jacket description says, that sounds like an awful cheezy gimmick straight out of Amelie or some shit; rather one guy has hiccups and we see the village go past him and mainly the camera just wanders off and watches various different people go about their day. There's not much of a plot to speak of, but it's actually a very easy watch, there's lots going on all the time, he doesn't challenge you with real time pacing.

There are a few really bad tasteless moments, mainly when the director uses modern technology / computer graphics for the growing plants bit, the airplane bit, the skeleton bit, all really tacky and out of place. There are also quite a few cuts and fades that are rather too cute; oh look he panned to a house and then cross-faded to a photo of that same house inside a bar, how fucking not clever and fucking film-school is that bullshit . It would work much better if the camera was just still and quiet and let the scene do the talking. Actually overall the direction/camera choices are really awful, but what the camera is seeing is so touching it works anyway.

I watched the three special episodes of "The Thick of It" that you can get on piratebay (it's sort of the spiritual predecessor to the movie "In the Loop"). I'd love to see the whole series but can't find it. At first I was worried that it would be too fast-talking and weird-British-accenty and impossible to follow, but after about 5 minutes my brain settled into the flow and it was no problem. It's quite excellent. I'm not sure it's a comedy, I don't think I actually laughed at any moment, but it's very engaging.

Someone loaned me this awesome old french cookbook "Bocuse dans votre cuisine". It's quite fascinating because it's so out of touch with what home cooks want to know, it's from an era when proper cooking was still considered to be this incredibly difficult precise thing. The recipes are really interesting to me because in order to simplify them for the home chef, he didn't abandon any technical rigor, instead he just made the dishes very basic. It's the exact opposite of what cookbooks now are like - modern cookbooks will give you lots of fancy ingredients that you mash together, and they will abandon all concern for doing things right. The Bocuse book literally has recipes for things like "haricots verts" with absolutely nothing added, zero embellishments, it's literally fucking "boil your green beans", but it's still a full page of instructions detailing exactly the precise way you should do each step. There are instructions like "wash quickly and gently in cold tap water" for each ingredient different amounts of washing and different care steps. Every other instructions is specified to be done "soigneusement" (eg. with attention, with care).

Inspired by Bocuse I present : How to refresh day old bread :

Step 1 : make sure you bought a proper loaf with a tender gluteny interior and a crusty exterior. If you didn't, then just throw away your piece of shit.

Step 2 : store overnight in a paper bag wrapped in a plastic bag. The plastic bag should be one of those crinkly plastic bags from the grocery store checkout, not a zip-loc or sticky filmy plastic bag. The paper bag should be closed loosely but not sealed. Leave on the counter away from sun and heat and moisture sources.

Step 3 : heat the oven to 400-450.

Step 4 : remove bread from bags and slice in half. Spray the interior with a fine water mist, just one or two squirts depending on the mister. The interior of the bread should be damp to the touch but there should be no visible water drops. Put the halves of bread back together.

Step 5 : (optional) apply a very thin coat of fat to the bread; this will make the crust crispy instead of just hard. If it's an italion loaf, use an olive oil aerosol can. For a baguette, use a very light spread of butter.

Step 6 : place in pre-heated oven. Watch through window for doneness. Desired doneness is to your taste, but generally the crust should just start to pucker and crack but not brown.

Step 7 : Remove from oven and let sit on the counter to cool. Do not separate halves of bread, it should be steaming inside.

Note that most "artisan" (lol) bread you buy in an American grocery store is basically day old already, so this should be done on the first day and then it should be thrown out.

The cupcake place (Cupcake Royale) on Capitol Hill sucks pretty bad. The cake is too dense and bland, and the frostings are too heavy and sugary and excessive. It just is not made soigneusement. And we still have no real bakery on the hill that makes actually tasty things like loaves of bread or croissants.

I have no problem with cupcakes in general as a food form, though we certainly don't need shops that *only* make cupcakes. A cupcake is basically the lowest form of cake. It's just basic standard fucking cake batter and frosting. That's like basic cake 101. If you went to an actual bakery, you would never order the fucking yellow cake with frosting because it's too boring, you could get the fucking almond-flour olive oil cake or the lemon curd meringue cake or something actually fucking interesting.

The recent explosion of cupcake places in Seattle is another real "WTF Seattle" moment for me. Seriously, cupcakes? First of all, all these places suck balls, they focus on the "nostalgia" and cutesiness factor instead of just making things that are actually good. But, more importantly, can you be more fucking Sex in the City ten years ago? Have you heard of the fucking Magnolia Bakery? Have you no shame? This is of course the city that is still in the middle of the tapas craze (I'm sure Arnie Becker from LA Law would be excited about this new fun "tapas thing" that everyone's doing). I'm embarassed for all of us.

" *** There will be a Mariners game tonight at Safeco Field from 7:10 PM to 10:10 PM."

This is the new fucking bane of my existance. If I was rich I would buy up all the Seattle sports teams and just disband them.

08-31-09 | New Bike 2.0

I redid a bunch of stuff on my new bike and I'm much happier with it now. It doesn't look nearly as cool, but it rides better and is more functional for me. I went back to plain old drop bars. I don't really care about being in the drops, but I need the shorter reach to the brakes, and I like having the curve of the drops to brace against for hard braking. I also swapped the stem to a shorter, taller stem, to bring the bars up and back to me now. I've always scoffed at people with tall stems like the Nitto Technomic (or worse, upward-angled stems) because it basically means you fucked up and bought a frame that was too small for you. Yup, that's what I did, I'm a tard, though the frame isn't actually too small, it's just too low and long. One thing that many people don't account for is that because of the angle of the head tube, when you raise the stem, you also shorten the reach. To get your bars higher and keep the horizontal reach the same, you would actually want to switch to a longer reach stem. There's also a weird thing going on in my case because my head tube seems to be 74 degrees and most stems are 73, so the stem is actually angled down a bit.

Sadly it doesn't look nearly so elegant and clean now that it's actually functional :


Some more little random building/tweaking stuff I learned for my own reference :

I don't like the standard advised way of putting on brakes. The standard advice is to hold the brake at the desired separation with one hand and then tighten down the cable bolt with the other. This is incredibly hard to do without slipping, and if you slip you get too much slack which makes the brake pads too far apart (proper brake pad installation has them just barely off the rim). Instead, do this : first back out the barrel tension adjuster about 10 times around (a lot). Now, hold the brake all the way closed, tight against the rim. Tighten the cable bolt. At this point the brake is way too tight, so screw the tension adjust back in until it's relaxed to the point that the the wheel runs free. I find this much easier and gives a very nicely adjusted brake.

Brake cable clatter. My bike runs almost perfectly silent, it's magic, it's like a ghost slipping by in the night. Because of that I hear lots of little noises that you might normally ignore and they bug me and I fix them. One of the last little bits of noise was the brake cable instead inside the brake lever bodies rattling around. To fix this, just wrap the brake cable end in a bit of electrical tape before feeding into into the lever to prevent the metal-on-metal contact. This is *very* hard to do after the brake cable is installed, so just do it first as a preventative measure when installing new brake cables. (not sure if this is a good idea on STI levers; I did this on old-school aero brake levers).

Loose spokes. Apparently my wheelset was shipped to me with the spokes rather on the loose side. I read a few other posts around the net that indicate this is pretty common, the commercial wheelbuilders tend to make the wheels at rather low tension. If you're a small rider this may be okay, but for a big heavy oaf like myself, the wheels flex which causes the spokes to knock on each other and make a rhythmic "ping" as they go around. (you can isolate the sound to this problem if it happens even when you don't pedal, and it only happens when the bike is under load). To fix, just tighten your spokes. I haven't found any good information about how tight spokes should be or how you're supposed to tell that they're tight enough.

Saddle rails. There's some weird shit with saddles and seatposts. Unlike almost every other bike part, they seem to just be made really sloppily. The actual spacing seems to vary between 42 mm and 44 mm. Looking at some new saddle rails you'll even see that they vary in separation from back to front. Apparently you're just supposed to force them to the width you need and tighten the clamps and it works okay. A bigger problem is that old saddle rails are 7 mm in diameter and new ones are 9 mm ("oversized" like everything these days for lighter weight and more strength). For the most part old rails will work in new seatposts, but new rails may have trouble fitting in some old seatposts. The advised fix to that is just to widen the old seatpost clamp with a metal file. Ugly. (on a semi-related note : there are a whole lot of seat post sizes ; my three bikes are 25.0, 27.0, and 27.2 so I can't swap any parts, yay).

08-28-09 | Phone Text Problem

I wrote a couple of posts ago about how I was having some mysterious problem with my phone not delivering texts. I've now discovered that the problem is sometimes the texts randomly go into "pending" mode when I hit send. I've discovered no rhyme or reason for this, I just sent about 10 texts this afternoon and later discovered that 2 of them went pending, so I'm sure the person I was texting with was rather confused by my lack of response and then sudden jump ahead in the conversation flow.

Apparently the advised fix ( 1 or 2 ) to this is to yank the battery while the phone is on !? WTF world. Straighten your shit out. (though that doesn't actually deliver your pending texts - you have to forward them to the original recipient still).

I've been stealing my neighbor's wifi for the last week, and I guess they finally wized up and turned on security. Damn it. I was being nice too, only email and a bit of web browsing, no downloads. Of course browsing the web is pretty damn bandwidth heavy these days. There really needs to be a "light" mode of the web that lets you just get text and only crucial pictures, no embedded media, but dear god that would require the web to actually be sane.

Now I have to decide between the fucking awful DSL that's slow and overpriced or the fucking awful ghetto cable Broadstripe that's in bankruptcy and has frequent outages and lack of service people. One of the things I hate the most in life is when I have to make a decision where I know both choices suck absolute balls and I will regret it either way.

It's kind of good not having it. Maybe it will help me break the ctrl-alt-M constant email checking and let me get down to real work. Except I need net to talk to the RAD perforce server.

Both Broadstripe and QWest have outrageous contract deals that make me angry. Broadstripe has a $100 sign up fee, which pisses me off to no end. Broadstripe also requires a 13 month term; that's so obviously intentionally designed to fuck over renters who often have 12 month leases, they know they can get an extra month of free money out of you. QWest has all these crazy complicated pricing models with asterisks upon asterisks. They advertise things "like $19.99 for internet" (*) but that's only with a qualifying phone package (*) (= that costs at least $35/month) and (**) only if you sign up for a 2 year term with big cancellation fees. If you just sign up for month to month with no phone, it's $70/month for the high speed internet.

I'm pro free market and all in theory, but it seems like the consumer and competition would be better served if we had some laws forbidding all these kinds of contracts. If you want to offer a subscription service it has to just be a month to month rate with no conditions.

08-27-09 | Oodle Image Compression Looking Back Pictures

I thought for the record I should put up some pictures about what I talked about last time.

First of all the R/D trellis quantization issue. Very roughly what we're doing here is coding to a certain bit rate. The "RDO" lets us use a smaller quantization bucket size, which initially lowers distrortion and increases our rate, but then we hammer on some of the values - mainly we just force them to zero, which causes some distortion and decreases rate; we choose to hammer the values that save us the most rate per distortion. (99% of the time all you're doing is turning 1's into 0's, so it's a matter of picking the 1 to squash to 0 which saves you most the rate).

Here are the results on "Moses" at 0.5 bits per pixel :

No R/D : RMSE = 9.9381 :


Unconstrained R/D : RMSE = 9.7843 :


You should be able to see in the R/D image that some of the image looks better, but other parts look much worse. The RDO has stolen rate from places where it was expensive in terms of rate to encode a certain distortion, and moved those bits to parts of the image where you can get more distortion win at a cheaper rate. This is awesome if your goal is to minimize RMSE, but it's unclear to me whether this is *ever* good perceptually.

In this particular case, the RDO Moses image actually has a worse SSIM than the No-RD image; this type of mistake is actually something that SSIM is okay at detecting.

In practice I use some hacks to limit how much the RDO can do to any one block. With those hacks I almost always get an SSIM improvement from RDO, but it's still unclear to me whether or not it's actually a perceptual improvement on many images (in some cases it's a very clear win; images like kodim09 or kodim20 where you have big flat patches in some spots and then a lot of edge detail in other spots, the RDO does a good job of stealing from the flats to give to the edges, which the eye likes, because we don't mind it if an almost perfectly smooth area becomes perfectly smooth).

Now for the hacky perceptual smooth DC issue.

This is "kodim04" at 0.25 bpp ; no RDO ; no unblock , no perceptual DC quantization ; basically a naive DCT coder :


Now we turn on the hacky perceptual quantization that gives more precision to smooth DC's : (unblock still off) :


Note that the perceptual quant of DC means that we are using more of our bitrate for the DC band, so we give less bits to AC, which means using a larger quantizer for AC to match the bit rate constraint.

Now with unblocking , no perceptual DC quant : (RMSE = 12.8565 , SSIM = 58.62%)


With unblocking and perceptual DC quant : (RMSE = 12.9666, SSIM = 57.88%)


I think the improvement is clearest on the unblocked images - the perceptual DC quant one actually looks okay, the parts that are supposed to be smooth still look smooth. The one with uniform DC quant looks disgustingly bumpy. Note that the SSIM of the better image is actually quite a bit worse. Of course RMSE gets worse any time you do a perceptual improvement. You should also be able to see that the detail in the hat thatching is better in the nonperceptual version, but that doesn't bother the eye nearly as much as breaking smoothness.

ADDENDUM : some close up pictures of Moses' waddle area showing the R/D artifacts better. You should zoom these to full screen with a box filter and toggle between them to see most clearly. You should see the RDO killing blocks in the collar area very clearly. All you really need to do is look at the last picture of these four and you should be able to see what I'm talking about with the RDO :

Portion of Moses at 0.75 bpp : No lagrange optimization :


With Lagrange RDO :


Crop of No-L :


Crop of RDO :


08-25-09 | Oodle Image Compression Looking Back

I did a little image compressor for RAD/Oodle. The goal was to make something with quality comparable to a good modern wavelet coder, but using a block-based scheme so that it's more compact and simple in memory use so that it will be easy to stream through the SPU and SIMD and all that good stuff, we also wanted an internal floating point core algorithm so that it extends to HDR and arbitrary bit depths. I wrote about it before, see : here or here . That's been done for a while but there were some interesting bits I never wrote about so I thought I'd note them quickly :

1. I did lagrange R-D optimization to do "trellis quantization" (see previous ). There are some nasty things about this though, and it's actually turned off by default. It usually gives you a pretty nice win in terms of RMSE (because it's measuring "D" (distortion) in terms of MSE, so by design it optimizes that for a given rate), but I find in practice that it actually hurts perceptual quality pretty often. By "perceptual" here I just mean my own eyeballing (because as I'll mention later, I found SSIM to be pretty worthless). The problem is that the R-D trellis optimization is basically taking coefficients and slamming them to zero where the distortion cost of doing that is not worth the rate it would take to have that coefficient. In practice what this does is take individual blocks and makes them very smooth. In some cases that's great, because it lets you put more bits where they're more important (for example on images of just human faces it works great because it takes bits away from the interior patches of skin and gives those bits to the edges and eyes and such).

One of the test images I use is the super high res PNG "moses.png" that I found here . Moses is wearing a herring bone jacket. At low bit rates with R-D Trellis enabled, what happens is the coder just starts tossing out entire blocks in the jacket because they are so expensive in terms of rate. The problem with that is it's not uniform. Perceptually the block that gets killed stands out very strongly and looks awful.

Obviously this could be fixed by using a better measure of "D" in the R-D optimization. This is almost a mantra for me : when you design a very aggressive optimizer and just let it run, you better be damn sure you are rating the target criterion correctly, or it will search off into unexpected spaces and give you bad results (even though they optimize exactly the rating that you told it to optimize).

2. It seems DCT-based coders are better than wavelets on very noisy images (eg. images with film grain, or just images with lots of energy in high frequency, such as images of grasses, etc). This might not be true with fancy shape-adaptive wavelets and such, but with normal wavelets the "prior" model is that the image has most of its energy in the smooth bands, and has important high frequency detail only in isolated areas like edges. When you run a wavelet coder at low bit rate, the result is a very smoothed looking version of the image. That's good in most cases, but on the "noisy" class of images, a good modern wavelet coder will actually look worse than JPEG. The reason (I'm guessing) is that DCT coders have those high frequency pattern basis functions. It might get the detail wrong, but at least there's still detail.

In some cases it makes a big difference to specifically inject noise in the decoder. One way to do this is to do a noisey restore of the quantization buckets. That is, coefficient J with quantizer Q would normally restore to Q*J. Instead we restore to something random in the range [ Q*(J-0.5) , Q*(J+0.5) ]. This ensures that the noisey output would re-encode to the same bit stream the decoder saw. I wound up not using this method for various reasons, instead I optionally inject noise directly in image space, using a simplified model of film grain noise. The noise magnitude can be manually specified by the user, or you can have the encoder measure how noisey the original is and compare to the baseline decoder output and see how much energy we lost, and have the noise injector restore that noise level.

To really do this in a rigorous and sophisticated way you should really have location-variable noise levels, or even context-adaptive noise levels. For example, an image of a smooth sphere on a background of static should detect the local neighborhood and only add noise on the staticy background. Exploring this kind of development is very difficult because any noise injection hurts RMSE a lot, and developing new algorithms without any metric to rate them is a fool's errand. I find that in some cases reintroducing noise clearly looks better to my eye, but there's no good metric that captures that.

3. As I mentioned in the earlier posts, lapping just seems to not be the win. A good post process unblocking filter gives you all the win of lapping without the penalties. Another thing I noticed for the first time is that the JPEG perceptual quantization matrix actually has a built-in bias against blocking artifacts. The key thing is that the AC10 and AC01 (the simplest horizontal and vertical ramps) are quantized *less* than the DC. That guarantees that if you have two adjacent blocks in a smooth gradient area, if the DC's quantize to being one step apart, then you will have at least one step of AC10 linear ramp to bridge between them.

If you don't use the funny JPEG perceptual quantization matrix (which I don't think you should) then a good unblocking filter is crucial at low bit rate. The unblocking filter was probably the single biggest perceptual improvement in the entire codec.

4. I also somewhat randomly found a tiny trick that's a huge improvement. We've long noticed that at high quantization you get this really nasty chroma drift problem. The problem occurs when you have adjacent blocks with very similar colors, but not quite the same, and they sit on different sides of quantization boundary, so one block shifts down and the neighbor shifts up. For example with Quantizer = 100 you might have two neighbors with values {49, 51} and they quantize to {0,1} which restores to {0,100} and becomes a huge step. This is just what quantization does, but when you apply quantization separately to the channels of a color (RGB or YUV or whatever), when one of the channels shifts like that, it causes a hue rotation. So rather than seeing a stair step, what you see is that a neighboring block has become a different color.

Now there are a lot of ideas you might have about how to address this. To really attack it thoroughly, you would need a stronger perceptual error metric, in particular one which can measure non-local patterns, which is something we don't have. The ideal perceptual error metric needs to be able to pick up on things like "this is a smooth gradient patch in the source, and the destination has a block that stands out from the others".

Instead we came up with just a simple hack that works amazingly well. Basically what we do is adaptively resize the quantization of the DC component, so that when you are in a smooth region ("smooth" meaning neighboring block DC's are similar to each other), then we use finer quantization bucket sizes. This lets you more accurately represent smooth gradients and avoid the chroma shift. Obviously this hurts RMSE so it's hard to measure the behavior analytically, but it looks amazingly much better to our eyes.

Of course while this is an exciting discovery it's also terrifying. It reminded me how bad our image quality metrics are, and the fact that we're optimizing for these broken metrics means we are making broken algorithms. There's a whole plethora of possible things you could do along this vein - various types of adaptive quantizer sizes, maybe log quantizers? maybe more coarse quantizers in noisy parts of the image? it's impossible to explore those ideas because we have no way to rate them.

As I mentioned previously, this experiment also convinced me that SSIM is just worthless. I know in the SSIM papers they show examples where it is slightly better than RMSE at telling which image is higher quality, but in practice within the context of a DCT-based image coder I find it almost never differs from RMSE; that is, if you do something like R-D optimized quantization of DCT coefficients with Distortion measured by RMSE, you will produce an image that has almost exactly the same SSIM as if you did R-D with D measured by SSIM. If RMSE and SSIM were significantly different, that would not be the case. I say this within the context of DCT-based image coding because obviously RMSE and SSIM can disagree a lot, but that axis of freedom is not explored by DCT image coders. The main thing is that SSIM is really not measuring anything important visual at all. A real visual metric needs to use global/neighborhood information, and knowledge of shapes and what is important about the image. For example, changing a pixel that's part of a perfect edge is way more important than changing an image that's in some noise. Changing a block from grey to pink is way worse than changing a block from green to blue-green, even if it's a smaller value change. etc. etc.

It seems to me there could very easily be massive improvements possible in perceptual quality without any complexity increase that we just can't get because we can't measure it.

08-25-09 | Linkage

I found a picture of myself on SqueezeChart which is a little bit disturbing. There's way too much representation of the modern Russian/PAQ tweaker crew on there. Personally I think anyone who doesn't release source code or a publication to their compression algorithm should be black listed. If you don't make your method public it may as well not exist.

Performance it's the name of the game. MC Spandx. This video confuses me because I can't figure out if it's making fun of racers or celebrating them.

Gravel Beach is a very cool blog by a beach scientist in the Seattle area. Lots of good pictures and info about beach accretion and geology and whatnot.

We watched Hard Ticket to Hawaii the other day. It's incredibly, it's perfect, it was far better than I ever imagined it could be from the famous clips : 1 or 2 . It's just so mind blowing on so many levels.

While up there at Scarecrow, had a peak at the awesome graffiti on Tubs ; ( more photos ).

Bald Man Watching is a nice blog of Seattle street art. Apparently I just missed "Atroleptic". WTF why do I always find out about things too late !?

A while ago I wrote about how I wanted to invent a backpack that was suspended off your back somehow so that air could get in and you wouldn't sweat so much. Well, guess what, that already exists and has for a while. If you go to any technical backpacking place like REI / Mountain Hardware / etc. you'll see lots of them ( For example ). They're quite awesome. There seem to be two main styles now. One is just a tight mesh that's soft and just held slightly off the backpack body by some pads without a hard frame. The other style is actually a hard frame that fully supports the backpack well off your back, and then has a cloth mesh sprung off the frame that your back rests on. The latter style lets lots of air in. Both seem very awesome for cycling. I haven't yet found the ideal cycling backpack. I would like it to be more variable sized. It should have drawstrings so that you can cinch it up into a tiny little ball when it's empty, but if you relax it all the way out it should be very large just in case you need to carry home a piano on your back.

08-20-09 | Moving

Moving makes you realize what's really important to you. You save out from the mass packing those handfull of things that you really must have. For me it was : laptop (LDO), bike stuff (water bottles and helmet), coffee gear (kettle, cone, cup, grinder, beans). Everything else I own can fuck off. That's all I need.

(Unfortunately the movers packed up my coffee gear which I told them not to pack so I had to spend half the night digging through boxes to find it).

I'm at the 15th Ave Online Coffee because I have no internet in my house, and it's really creepy here. The vibe is so bizarre. It's eerily quiet, but disturbed by the very loud hum of the refrigerators.

It appears the cable providers here no longer offer anything but digital cable (you can only get "basic" with non-digital cable). I fucking despise digital cable. I like my TiVo, I don't want your damn DVR box. Digital cable also takes longer to change channels, and while overall the signal quality is better, it has much worse artifacts when there's a signal glitch - it tends to drop out completely or turn into pure noise instead of degrading gradually. I'm tempted just to go with no TV at all and use the internet, but for regular shit like "The Daily Show" or cooking shows it's too much pain to download them all. Real TV series (ala The Office / Dexter / whatever) I'm perfectly happy just getting off the computer. Apparently the cable in my neighborhood is Broadstripe which has ridiculously bad reviews (and also recently went into chapter 11). Another option is DSL + DirecTV, which I also don't want. Waaa I just want internet and Comedy Central waaa.

The Russians or Chinese really need to offer a TV torrent subscription service. They could just set up a bank

My new house is so big and empty. I don't really like having two stories. It's way bigger than I would like, but c'est la vie. It's also so quiet it's kind of strange. I know, I know, I'm a big whiner, I'm not actually complaining about how quiet it is here, it's just kind of a shock. I actually really like living in the heart of the city with people going by and noise during the day, I just want it quiet at night when I'm trying to sleep. In theory it's possible if houses are designed right (living room on the street size, bedroom on the back side, and the back side is treated as a quiet zone by all residents) but in practice that never works.

08-19-09 | Bike Ride Thoughts

I can't stand biking with people who are all type-A about it. They track their cadence and their heart rate, they train in tapers, they measure their blood sugar. It's all so analytical and thorough and careful and uptight and exactly everything that I'm trying to get away from when I take off on the bike. I want to ride away from computers and work and being right and worrying about every little thing. I ride out until it's just my legs and the wind and the road and we see what we can do together.

My favorite part of the ride is around mile 20 when I start to get a little light headed, and the monotony of the rode has murmurred to me like a mantra "pedal pedal pedal pedal" lulling me into a sleep-like zen state. Then everything disappears from my head and there is only breath and legs and spilling pedals and the road flying underneath. At that moment I can see the world without analyzing it, without wanting anything from it, without seeking anything in it. Normally when you're driving around you're looking for your destination, or even just looking for good views, or just looking at the houses and judging them, evaluating them - your analytical mind is going over everything you see, you don't just see it, you see it and process it, your mind attacks it, takes it apart. In the zen state I reach a point where the world is just going by and the analytical mind turns off and the world is just *there*.

People who are all analytical and careful about how they ride are like people who want to be really clean and proper during sex; no! this is the time to be loose and wild and relax and just go with what your body tells you to do!

Biking and bike racing are all about managing your self. You have only so much energy, you need to decide how to use it. If you burn it all up early you'll bonk and have a terrible ride. Over time you get to know your own body, you know you can sprint up hill A without going into the red, but not if you're having an off day. If a big hill is coming up, you know you have to take it easy for a bit to recoup so you can have energy to take it on. I think all these same skills are very important in life and work. You need to be aware of yourself and what drains you and how much energy you have, and how to prepare for things.

It occurred to me that one of the big things I do when I'm in dating mode is I very carefully manage my self. I try to make sure I'm in a good mood and have energy when I go on a date. I don't just rush straight from work or a hard workout to a date when I'm all exhausted and in a bad mood. I'll avoid making dates when I know I'd have to go through traffic to make the time, because then I would arrive all agitated and pissed off. Sometimes I'll push into the "red zone", forcing myself to be energetic on a date when I really just want to go to bed.

I stop all that once I get into a committed relationship. I know in my past I've been disappointing to my lovers because I'm a lot of fun during the dating phase, we do lots of new stuff, I'm talkative and energetic. Then over time as we settle in together, I stop doing all that because it's a fucking pain, and I'm just more myself, which is a boring homebody who hates to talk. I know you need to keep doing things together, so I'll still find things for us to do and we'll go out on "date nights" or whatever, but it's not the same. I just realized a big piece of the difference is this managing myself issue.

08-19-09 | Done Bike

I got the cables on, and after a few more stupid mistakes and redos, it came together pretty fast. The finished bike :



(yes the bar tape is fucked up because I had to take it off and redo it and had already cut it; I have to order new tape and do it anew).

Some links I used :

Sheldon Brown - Cables
Sheldown Brown - Derailer Adjustment-How To
Park Tool Website - installing new chain
Park Tool Website - derailleur overhaul
Park Tool Website - Cranks
Park Tool Website - cables
Park tool - bar wrapping
good - how to adjust headset
Bicycle Repair Washing Cleaning Your Bike
The Compulsive Soul's Guide to Cleaning the Bicycle

Some random things I've learned :

Having a clamp on front derailleur instead of a braze-on is kind of nice because it gives you infinite easy adjustment capability. However, it means the front derailleur clamp needs to be *REALLY* tight or it will slip. Even very tiny slips will fuck you up because the FD cage needs to be right next to the chainring, so even an 0.5 mm slip is no good. I read some advice on the net to put a piece of electrical tape inside the FD clamp to keep it from scratching the downtube. This is terrible terrible advice. The tape makes the FD clamp slippy which makes it impossible to get a good adjustment.

Some good advice from the internet : for wrapping bar tape, inside out electric tape on the bars is awesome. Take some electrical tape and wrap just one loop around the bars in the curving part, wrap it very tight so it stretches. Now when you put on the bar tape it will hold in the corners much nicer. Another good piece of advice is to work your hands around the bar tape in a rotating fashion like you would when riding to get it to tighten up before applying the end piece of tape. Also in general, really pull the tape very tight as you wrap, you want to really stretch it as you put it on.

If you want to seal on electrical tape really clean and tight, you can use a little heat - even a hair dryer will do the trick. It gets the tape a tiny bit melty and it contracts and shrink-wraps and gets rid of any little bubbles or folds. Warning though, it makes it much harder to remove.

Once you have all the tools, putting together a bike is really very easy. I don't think there's a single hard step; even getting all the cable adjustments just right is pretty trivial. The guide sheets that come with Shimano gear are actually really excellent. I always consult my two bike books plus the internet to get many opinions, but I could've actually just used the product installation sheets and been fine.

You can't cut cables and housing with basic wire cutters, but any good quality long-lever wire cutters will do the job just fine. You don't need a fancy "cable cutter" tool, just use high-leverage cutters and make a quick hard cut. Once the cut is made, if you want everything to be just perfect, a dremel rotary grinder does a really nice job of cleaning up the cable ends. A push-pin makes a perfectly good "awl" to open up the cables (mainly an issue because the dremel will melt the outer plastic).

A work stand makes life so much easier and is totally worth the $100. Even if you're just servicing bikes and not building, I highly recommend it.

The correct chain size seems too tight! Don't worry, trust the Park Tool / Sheldon Brown advice of going big-to-big plus 1 inch. You will think "no way is this long enough to work once it goes through the rear derailleur". Yes, in the big-to-big position this will pull your rear derailleur really far forward - that is the intent! You should really never be in that gear combination, the goal is just to make it *possible* to get into that gear so that you don't break the bike if you do it by accident. The more important thing is that when you're in the small-to-small gear the read derailleur doesn't coil up right onto the sprockets so that the jockey wheels almost touch the sprockets. The rear derailleur is meant to be tugged out under tension, not slack and balled up.

Ferrules don't seem to go into the brake barrels. I tried to get them in at first and was dismayed, but then I noticed that both of my other bikes don't have them either. It seems the brake barrels are made so small that you can only just get the cable housing in, the ferrules are too fat. Ferrules do work of course at the lever end of the cables.

It seems to me you can adjust a headset just fine with only one headset wrench. I don't have two so I just did it with one and it seems perfect and tight to me. We'll see if this comes back to bite me.

Oh, there's some clusterfuck with saddle rail spacing. I didn't even know this, but there appear to be two different sizes. I'm not even sure what they are because I can't find anyone else talking about this, but I think they're something like 42 mm and 44 mm - which is too different to be compatible. You must have the right seat post for your saddle. If you look at pretty much any saddle online it won't say anything about the rail spacing. I assume if you buy a new saddle and new seat post they will match, but if you are trying to match an old-to-new you may have a rail spacing problem. If anyone can find any authoritative info on this issue I'll add it. (I'm not talking about the rare 30 mm very narrow Keirin NJS rail spacing).

Don't cut anything or put cable crimps on until you're sure everything is right. When you wrap the bar tape, just temp tape it down and leave the rest hanging off at first. When you thread the cables, just leave all the excess length hanging out and bolt them down. Then go through and check everything and adjust it. You can even do all your derailleur adjustment with the cables just long and flopping around. Once you're sure it's all good, then cut and crimp. Once you crimp, you can't run the cable back through the mechanism to get it out, so you can't make any big changes if you discover you did something wrong.

And in general about the bike. The good :

Holy shit the shifting feels good. Better than any new bike I've ever ridden in a shop. I assume this is partly because it's all brand new Ultegra, and unlike most shops I didn't try to save money on cheap cables or cheap housing or anything. Maybe partly because it's down-tube shifting which does give you a shorter tighter line. I also assume it's partly because I did everything slowly and carefully and tried to get everything just right. There's no slack anywhere, you just think in your head that you want to shift and it's already done and it's whisper quiet. Amazing.

The compact double front chainrings are great. I thought they might make the front shifts rougher because it's a much bigger step for the chain, but I see no problems at all with it. It gives me a much easier gear for the steep hills here. I'm very happy with 12-27 on the back too.

It feels crazy fast and nimble. Mainly I think it's because the drive train is so stiff, there's no play in the bottom bracket or the chain, you touch the pedal and you instantly go without all the mush I have in my old bike.

The bad :

I made a huge mistake. The geometry is not what I thought. I tried to be really careful ordering the frame, getting all the sizes right and checking angles, but somehow the frame is not what I expected. The way bikes are measured is really insufficient, the main measurement used is the seat tube length. So my new frame has the same seat tube length as my old bike - but the bottom bracket on my new bike is much lower to the ground. The result is that the top of the seat tube is much lower, so the top tube is lower, the whole bike feels lower even though it's the same "size". The size anomaly comes because the angle of the chain stays is a lot steeper on my new bike - they go up a lot to attach to the wheel, while on my old bike they are closer to horizontal. Then the fork on the new bike is shorter, and the wheel is much closer to the down tube. I'm not happy about the geometry change, I thought I was getting the same geometry as my old bike (which seems impossible to find in new bikes sadly; I don't understand why classic road geometry has been so abandoned, it's totally the way to go). I think that I can switch to a really short stem (60 or 70 mm) to fix the geometry, but it's a rather annoying issue.

I'm not loving the bullhorns. The main complaint I have is just that they're too long, you have to stretch too far forward to get to the brakes. I guess they're not really meant for the brake usage like I'm doing, but a lot of people do it these days, it's not good. If you compare to how far forward you would be on the hoods on normal drop bars, it's a good 'nother two inches to get to the ends of the bulls - and the bullhorns I have ( Nitto RB-018 ) are just about the shortest you can buy, most of then are even much longer which would be insane (and many of them are actually drop-down to make you lower which is also nuts). The flats on the sides don't look that long when you have them off the bike, but they're actually way longer than you need, you could get away with a few inches less. I think I might go back to plain old drops. (on the plus side, I saw some forum posts saying it was hard to get bar end brake levers in the RB-018, which seems to just be nonsense, the Soma Pursuit levers go in just great).

The Selle Italia Gel Flow saddle ain't got no gel. I think it would be an okay saddle for use with padded big shorts, but it's too hard for city riding that you do without padded shorts. Amateurs don't know this, but serious bike saddles are intentionally made very hard because you're supposed to have the padding in your shorts. It's stupid to ride a hard saddle on a city bike though, because you're just riding directly on the hard. On the other hand, the super-padded saddles made for the casual market are even worse. There aren't many good saddles that are made for serious riding without padded shorts. I might try a Specialized Body Geometry Avatar. On the plus side, I did find the Specialized sizing advice based on measuring your sit bones to be pretty good.

Anyway, I'm gonna try riding it for a few weeks and see how everything feels and then make a few changes. Right now I'm thinking go to a shorter stem to reduce the reach and just go back to normal drop bars. The handlebar that I want is just the top part of normal drops and brake hoods with no actual bottom drop part. Maybe I'll get that custom cut.

This was always intended as a learning experience, but it got a bit out of hand and turned into a serious build. Part of the problem is there's such a small price step for incrementally better stuff, it's very tempting to upgrade each piece. Like a full 104 grouppo (without STI levers) is maybe $400, while full Ultegra is $550. Of course you should upgrade. A cheap saddle is maybe $75, a great one is $150, well that's your butt and gonads, it's important, upgrade. Cheapo wheels are maybe $200, awesome Mavic Open Pros are only $300, of course you should upgrade. And before you know it you spent $2000.

Oh well, I'll do better on the next one.

In general I really advise against trying to buy a frame on ebay. Yes, you can find great bargains, but people don't measure things right, and measurements are crucial. Usually they will only post a "size" (which is a seat tube length). But is that the actual seat tube length c-c (center to center) or c-t (center to top) ? Or is the manufacturer's nominal size which may not exactly match the c-c or c-t seat tube lengths? They never post other crucial sizes, and if they do they're probably wrong. Pretty much the only way I would ever buy a frame on ebay again is if it's a frame where I know the exact model and year and manufacturer's nominal size, and I can look it up in the original catalog and find all the measurements. Even then I might have to cut out a cardboard replica of the frame to see it in person before buying. And by then the auction is over or somebody else insanely overbid it out of reasonable price range. It's just not worth it.

08-18-09 | Tuesday

Broken emails? I've heard some reports that emails weren't getting to me while I was gone. If you sent me an email over the past few days and I didn't reply to it (eg maybe I didn't get it), can you send it again? You can CC "cbloom.cbloom" at google-mail

When I was sitting in the Philadelphia airport I noticed one of those wifi "honey traps" that people talk about. I foolishly took a look at the wifi list to see if there was anything available. The list was something like "attwifi", "strbks", "Free Public Wifi". Hmmm, guess which one is just begging you to use it? The "Free Public Wifi" showed a different icon indicating it's a direct peer to peer link or something, not a broadcast access point.

Ugh. I'm really unhappy with Visual Assist X. All my fears about moving from VS 2003 to 2005 and the pain of having to get on VAX are coming true. Not only is the usability of VAX much worse because of the fucking awful suggestion box that they refuse to fix (you can see my Whole Tomato forum thread about it here in which they make up ridiculous reasons why it "has to be" the way it is; umm, no it doesn't, just make it like VA.NET).

This photo from Vintage Seattle is just proof that in every way people were way more awesome in the past. Nowadays people would be like "whaaa the shop is dirty; whaaa that's a big scary looking dead animal" ; in the past they were just like "pass my sword, bitch, I needa cut me some lamb, then we're gonna fuck and smoke and punch babies".

I did the inspection for my new house today. The owners are total nits about minor damage, which is really fucking annoying. It means I have to be crazy careful about noting all prior damage. And they sprung some crazy bullshit on me. Part of the reason I took the place originally was because the kitchen has these great huge butcher block counter tops. Today during inspection they hand me a lease addendum sheet that includes a bunch of extra conditions for care of the place, one of which is "do not cut or place hot objects on the kitchen counter tops". Umm, WTF !? Do you understand WHY people get butcher block counter tops? It's not because they look pretty and quaint and rustic, it's so you can fucking cook on them and not worry about. They're supposed to get scratched and water-marked and heat-stained, it's part of the charm. I cannot fucking stand people who are anal about keeping their kitchen immaculate. Your stove should have burned-on food, it should have black heat marks, your counters should have food stained grout, it should all be delightfully USED.

Anyhoo, I haven't signed the lease addendum yet. I think I might refuse. In any case, I think I just decided while I was writing this that I'm just gonna ignore those instructions in any case and fucking do whatever I want to the counter tops.

Poker blog from Brian Townsend is a nice read for me. Of all the big online pros, BT (sbrugby/aba) is the one I relate to and respect the most. For reference, he was the superstar of 2007, the first really huge online kid big winner. He won around $6 M in 2007 playing NLHE at the highest level. Then people finally quit playing him in NLHE (and in fact, all of the biggest money NLHE games dried up around that time, because it became too clear that a few amazing players could easily dominate them, and because Guy la Liberte quit playing so much). In 2008 the big money games switched to PLO and BT tried to follow the big money and play PLO at the highest stakes even though he had no expertise in it. He quickly lost around $5 M, mainly to Europeans like Antonius who had PLO experience, but also to Ivey who came on the scene and started dominating everyone. But unlike most of the crazy online kids who just gamble and go through huge swings, BT took his lumps, withdrew his remaining money, and started over again, grinding up PLO from the lowest stakes, learning the game, slowly getting better, moving up levels as he got better, and in 2009 he made it back to the biggest PLO games in the world and won in them again.

08-15-09 | Regenesis

I'm obsessed with the concept of tearing things apart to rebuild them anew, rebuild them fresh, with new ideas, and without all the old problems.

The classic one is just tearing up your life from time to time; change jobs, move cities, don't talk to old friends, tear everything up and rebuild. The fantasy is that you can start over without all the baggage of how people know you and the history you've laid down, you can build up a new you the way you want to be now. It never seems to work out.

I love the Lance Armstrong myth, even though it has only a small basis in reality. The myth is that before cancer, Lance was a very mediocre cyclist. The disease and chemo destroyed his body, wasted his muscles, made him emaciated, and turned him into a blank slate. Then when he got into recovery he resolved to become a great cyclist, and went into training hard. Any normal human who becomes a cyclist has all the baggage of muscles that were used for walking and playing catch and all the normal things that you do as a kid growing up, but Lance had shed all that to the disease. Thus as he trained up his musculatural, putting pounds of muscle on week after week by riding the bike, he built up a body that was specifically built for cycling. The myth is that only through the disease tearing him down to nothing could he rebuild the super-human cycling metabolism and physique.

It's not true, but I love it. (almost all of the great sports stories are myths that aren't quite true, that let us turn sport into parables, we constantly make up David vs. Golliath myths in sports, or Agincourt longbowmen vs the French, etc.)

Working on this fucking physical therapy every day that seems to never get me anywhere, I dream of tearing down. I imagine that if I could just wipe the slate and rebuild, I could easily fix everything. I actually think that's true in my case; my biggest problem now is that my neuromuscular system has picked up some very very bad habits over the years. Because of my shoulder injury, my long thoracic nerve developed "guarding" which basically means it doesn't fire the serratus. It's perfectly capable of doing so, and my serratus is plenty strong, but my brain doesn't tell it to. Furthermore, years and years of horrible computer use posture mean that whenever I stop thinking about it I slip into a nasty hunch, and even if my back doesn't hunch, my shoulders try to creep forward and up reflexively anytime I need to do something with my arms. If only I could tear down and rebuild, move out of this body into a bowl of stem cells and start anew.

08-14-09 | Travelling

I'm on my way to New York. I was thinking about not taking my lappy, but at the last minute I caved in. If I was more confident that this would be a pleasant trip I would have left it, because I do take great delight in just getting away from the computer and the internet hive from time to time, but I'm worried this trip is going to suck balls and I'm going to have to hide in my room, and then I will desperately crave my lappy.

Typing on my actual laptop keyboard is hurting my wrists already, and I can feel it wreaking havok on my shoulders and neck. If you regularly use a laptop, be aware that you are intentionally destroying your body.

ADDENDUM : for some reason my email isn't working, so if you're sending me mail, I can't respond. I'm sure I could work it out but I'm on vacation so I'm not dealing with god damn internet problems.

I'm sitting on the plane next to an old couple. The woman was reading "People" ; she was skipping the articles, because lord knows the articles in People are far too serious to bother with; she seemed to mainly be interested in the pictures of celebrity couples, who was dating who. As if that wasn't bad enough, the husband would occasionally lean over and say "now that's a fine looking woman", invariably in regards to some 20 year old starlet who has absolutely no redeeming physical characteristic other than that she's professionally made up and 20 years old, at which point the wife would scowl silenty and flip the page.

I would travel a whole lot more if flying and airports were more pleasant. Like if I could take a plane in the Pan Am era, I would just travel whenever I have free time. As it is, travel makes me so miserable that I feel like I need at least a week for it to be worth the misery of a travel day on each end. And of course, flying is only very miserable if everything goes completely according to plan; there's a high probability of cancelled flights and lost luggage, which turn the very miserable into nearly-suicide-or-homicide-inducing. If I could teleport, I would go to all kinds of crazy places, because I actually adore being in foreign countries and off the beaten path places where you have to mime what you want because you have no common language and all that.

There's an article in the September Atlantic about health care. It's quite good, it summarises my own viewpoint quite thoroughly. I think the problems with health care are very obvious (though obviously not obvious enough, because almost everyone gets them wrong - it has nothing to do with universal health care or single payer or electronic records). Perhaps the most important problem is the whole idea of using insurance to pay for things that aren't just catastrophic accidents with huge costs. Insurance in general is a horrible thing which is only to be used as a necessary evil. Sadly consumers are unbelievably fucking retarded, and they would gladly severely overpay for a subscription in order to get "free stuff" rather than just pay for each item/service individually out of pocket. While I think the Atlantic article is very good at getting to the crux of what the real problems are, it is (like me) light on solutions. His HSA suggestion is an okay start, but the more you think about the details, the more problems arise. How exactly do you encourage people to pay for preventive care, how do you subsidize people who can't pay enough into their HSA themselves? How do you motivate people to control their costs once they are already paying beyond the out of pocket maximum? It's also all completely irrelevant mental masturbation, because we have zero chance of actually doing anything serious to reform health care.

It's quite ridiculous the way the Republicans are reacting to health care reform. For one thing, the proposed bill does almost nothing. For another, their stance now has basically become that of an ultra-liberal Johnson democrat. They are demanding infinite care health care for all. God forbid you ever control costs on the eldery! I mean, of course it's ridiculous that all of their attacks are just completely fictitious made up nonsense, but even if you ignore that, the positition that they're taking on the made up bill they're attacking is quite strange. I really think the Republican party basically has no political identity any more except "people who are not in NY or CA and are angry/jealous that NY/CA dominate the US".

I'm in "The Hamptons" now and it's really one of the creepiest places I've ever been (similar to Carmel or Vegas). You've got all these rich New Yorkers who dress up in the uniform of white with ridiculous name brand sun glasses and strappy shoes and such and try to pretend like they're the "it" people. Then they're surrounded by the locals and service people that are the total Long Island cheeseball guido stereotype - muscley guys with berets and pouty lips and dark tans. Everything is so incredibly self conscious, and everyone's idea of what's good is entirely based on what other people are saying is good; it's the kind of place where The Emperor's New Clothes could literally happen. On the plus side, the beaches are actually really nice.

I met a girl in Seattle just before leaving and I'm quite smitten; I can't get her out of my head. I wish we were both unemployed so we could just ride around the city all day, go hiking during the week when it's not crowded, take long road trips, travel, and just do whatever feels right at each moment. When I was unemployed in San Francisco I really wanted to meet a girl who was unemployed so we could just run around together and be free. I did meet a few, but most unemployed people are either huge losers and flakes, or they're all stressed and obsessed with finding a job, which is not fun to be around. It's hard to find someone who was in the situation I was in, being unemployed but having a bit of money and just enjoying it and not being a total hippie/hobo wanderer.

I've developed this really bad habit of just checking my email reflexively any time I'm sitting at a computer and have a lull. So as I'm writing this, each time I take a pause, I hit ctrl-alt-M which fires up my Eudora, and I'm not on the net so it just shows error boxes. I close it and come back to this and then a few seconds later, boom ctrl-alt-M just fires again. For one thing yikes my muscle memory for key combos is strong. But more importantly that's just a really bad kind of habit to have. I might have to pull a Casey and cut myself off from email and web browsing for a while. It's become a crutch that I use to feed the ADD and save myself from painfully intense focus.

My brother thinks you should spend money to have fun any time you can, that memorable experiences are worth almost infinite amounts of money, so no price is too high. So, you know, thinks like a $250 helicopter tour of Hawaii or a $150 concert ticket to see Radiohead or whatever are all good deals. I sort of agree in theory, but in practice I find that those expensive things often aren't actually that much fun. For example, I find that most $5 concerts in small sweaty hipster clubs are better than big stadium shows for major bands. A lot of fancy hotels are just weirdly stuffy and unpleasant to be in, and the beds aren't even that great. Expensive tours are often really annoying with lots of waiting around. I have been trying lately to just be more liberal with money and try to buy happiness whenever I can, but I find it really hard to find. There are a few cases where I think it's worth it - I'm never cheap with food any more (though a $2.50 Vietnamese sandwich is usually better than a $50 nouveau bullshit CIA sear-and-sauce faux-fancy meal), and I think the next time I fly I'll go ahead and go first class, or at least go Virgin America if possible. This fucking plebian cattle car flight bullshit is not worth the savings.

I do like my brother's philosophy of trying to make trips more memorable, more interesting, more exciting. Don't just go visit the Grand Canyon and look at it like every other bum - take the mule ride down to the river, or raft it, do something. Do the shit everyone else wishes they were doing; why not?

The fucking tards on the airplane are all annoyed when they have to wait, they start huffing and stamping their feet like agitated cows, but then when it's their own turn to go through security or get their bags off the plane, they don't have their shit ready to go, and they take their sweet time ever so slowly going through their stuff. It's fucking hypocritical and inconsiderate and retarded. Par.

BTW this is yet another case where I'm not even talking about expecting people to actually be good. Expectations are so far into the gutter I just want them to be halfway reasonable and they still fail. Really you would like people in the airport to be quick and efficient and considerate, but without getting all uptight and impatient, they should still be relaxed and friendly. Lol. It sounds silly to even state what good people should be like because it never ever happens. So, now we lower expectations. If people would just pick one bad way to be and be self-consistent, I could tolerate that, like if they were all antsy and uptight and in a rush, but they actually went fast and had their shit together, that's fine, or if they're slow but they're actually relaxed and friendly and don't give you passive aggressive looks when you have to take time for something perfectly reasonable, that would be fine too. But no, they have to fail in every possible way and be both impatient dicks and also incompetently slow. Who the fuck doesn't know that you need to have your ID in your hand at the fucking security, and take off your damn shoes and fucking get your shit together. The fucking TSA should just taser people who are too retarded to go through security quickly.

08-13-09 | Bike progress

Step 1 : get bike in the mail and take it apart :

new bike

Step 2 : realize you have to take it to a shop to get the headset changed; get back bike, realize you don't have the tools to pull the bottom bracket, so take it to another shop. Clean and degrease. Sand off rust. Spray with frame saver (rust proofer). Clean again to remove excess frame saver. Touch up paint chips. Wax. All clean and naked :


Step 3 : wait for more parts to come in the mail. Realize some of them are wrong and have to send them back. Realize you don't have the tools to put on the new style bottom bracket (*), so buy those. Finally once everything is in order, you can put the parts together (this step is actually really easy) :


It looks like a finished bike now, but it has no cables on it yet, which is the hardest part (that's when you do all the adjusting). I also realized that I started wrapping the handlebar without the cable under it, so I have to redo that :( I was kind of thinking of running the brake cables in the old fashioned external big loop style, but after seeing the bike put together I don't think that's a good idea, and it doesn't work with the pursuit levers anyway.

(*) the new Hollowtech II bottom bracket is really strange. It requires yet another whole new set of custom tools (sigh). 99% of the difficulty with working on bikes is every generation is different and requires different tools. The new bottom bracket is very strange, it's not an axle supported by ball bearings, instead it's just a cylinder stuck into a larger cylinder. I guess the larger cylinder actually has a ring of bearings in it so it winds up being the same thing.

Also, you can't really see it in the shitty pictures I took, but the bike has a Shimano Ultegra Stem with hidden binder bolt from Peter White Cycles . This is just about the sexiest piece of bike hardware I've ever seen in my life.

08-11-09 | Blogs

Hideous Belltown is pretty excellent; definitely in the snarky cbloom / Surly Gourmand vein.

I've been reading through Maciej Ceglowski's Idle Words which is just full of gold. He's got a great post on Paul Graham which leads me to another review :

I fucking despise Paul Graham. He's the Malcolm Gladwell of bloggers. Every time somebody finds out I'm a programmer and I write a blog, they ask if I read Paul Graham like they're letting me in on some secret gold. For some reason this sort of vague trite cutesy anecdote writing is very popular with the semi-intelligent set. These guys tell some little story or make some stretched metaphor, and wind up making a point which is either just wrong, or completely trite and uninteresting. Paul Graham commits the added sin of being one of those fucking dot-com luckboxes who wrote some fucking lisp scripts at the right time and now thinks he has all kinds of business insight because he was successful. There's nothing worse than the condescending "words of wisdom" from some jackass who luckboxed their way into riches, it's like when some awful poker player happens to beat you in a tournament and then wants to talk about their clever strategy of playing super loose and passive in order to see more showdowns.

On a semi-related note, I enjoy reading the page 2 interview in the NYT sunday business section because it is just such amazing comedy gold. Every week they publish an interview with some CEO where they ask them about what management lessons they've learned and their secrets of success and whatnot. Every single seek without fail it's a stream of platitudes of the most ridiculous order. You learn business gems like you should hire good people and make them happy, communicate personally and give them clear direction and strong leadership. Wow! Now I can be a CEO too!

And this just in - sweet jesus, is The Surly Gourmand actually ME in disguise? Maybe so.

Some links :

Sebastien Tellier is ridiculously awesome.

Old Road Bike FAQ has some gooders.

Why Should Engineers and Scientists Be Worried About Color? ; old article from IBM, very good.

Some random dude's photos

08-10-09 | Information Sharing

I'm really quite upset by all the new ways of "sharing information" that are really just information obfuscation.

The thing that makes me so upset about Twitter is that I know interesting people who are posting interesting things on twitter - but it's just a shitty unsearchable, unclarified version of what they should be writing. If you have a thought that's worth sharing, it's worth taking a second to flesh it out and write it in at least semi-proper sentences.

Everything you spew into the interent is getting recorded forever and adding to our pile of information overload. Take a second, make it worth everyone's time to read.

Part of the problem is everyone jumps on new tech trends even when there's no need to. Text is a wonderful thing. It's searchable, random access, printable, I can snip out the bits I want to save or re-read.

I don't even like audio/video for information presentation. I've been meaning to watch Casey's video on GFK for like two years now, but I have to actually sit and watch it for AN HOUR. One of the amazing wonderful things about this great technology called "text" is that you can skim past the parts that are boring and that you already know, and then when you get to something tricky you can stick on it for a while or read it over and over, etc. Text is user-pace-controlled which is wonderful. Text is curteous to the client. Text compliments the intelligent of the client, it lets them make their own decision about they consume the information.

Slides of course are fucking garbage. It's hardly even worth downloading them; hell even going to talks at conferences is of very limited value. It's almost always better to just read a paper if there is one. In fact, conferences would be a hell of a lot better if they published the papers first and everyone read them *before* the conference, and then you could have a Q & A with the author about topics that are beyond what's in the paper, instead of just having them stand there are tell you about key points in the paper.

Today, it's Mike Acton's sketches on concurrency that are making me mad. It looks like there's some interesting stuff in there, but WTF Mike, this is like a Andy-Kaufman-esque performance piece indicting the ridiculousness of modern internet information sharing techniques.

I think in many ways the internet would be a better place if it was still plain text. Yes I am a luddite.

08-07-09 | Frizzle Bizzle

When you buy a giant SUV, you relinquish your right to weave around in traffic and try to cut ahead of people and change lanes abruptly.

VS2005 has this fancy import/export settings thing. Hey, cool, that's nice, I can use it to keep the settings of my various machines in sync. Except that it doesn't work. It doesn't import/export settings for addins. Hey, that would've been pretty easy to support but I'm not surprised, I'll let that slide. It also doesn't import/export the "VC Directories". Hmmm now I'm left wondering what else it doesn't import/export.

I despise that if I even accidentally touch my mouse to a disconnected network share drive, the computer stalls for like a minute.

One of my compact flourescent bulbs burned out. WTF am I supposed to do with this thing? I'm guessing 90% of people just throw it in the trash.

Some damn hobo stole our glass recycling bin. They've been coming for quite a while and taking glass out of it, I'm fine with that, though I wish they'd do it during the day instead of in the middle of the night, because it's quite noisy. But taking the whole damn bin is not cool. Curse you, hobo.

I have a lot of problems with text messages; people claim to have never gotten texts that are definitely filed in my "sent" folder. Texts to and from me appear to have random super-long delays sometimes, like occasionally even hours. For the communication medium that is our new standard, it seems to really fucking suck. Of course it may be partly because of my ten year old phone, but I've observed other people have these same problems.

The grey skies all this week have been a horrible foreshadowing of the winter to come when it will be nothing but grey and sadness all the time.

In college I used to do this thing called the "neverending juice". I would make a pitcher of frozen juice, then when it gone down to 1/4 - 1/8 remaining, I'd put in a new frozen juice thing of a different flavor, then when that got down to 1/4-1/8 I'd add another. Each generation, the flavor would get more complex. I suppose in theory it was mildly food-poisonous, but I never got past 5 generations.

There's this fantasy (often exploited in movies) that you can get over relationship problems by just getting in a big fight and yelling the things you really feel at each other. In reality, that would be a huge disaster, because in my experience at the core of all relationship difficulties are some fundamental core problems that can't be solved, and you just try to pretend they're not there and not talk about it. The actual yelling of truths would be things like "I just don't really enjoy being around you!" or "I think you're only with me because you're broke and I have money!" or "You're getting old and ugly and I'm not attracted to you any more!" or "You're an insecure coward who covers it up by trying to act tough!" or "your complete lack of life outside of me and work makes you boring and needy" or "you make yourself feel smarter by making others feel dumb" or "you're not committing to our relationship because you think you're too good for me", etc. When somebody says something to you that you know about yourself and hate, it doesn't help anything, it makes you feel awful, and you can't actual fix it, because in reality knowing is absolutely no part of the battle.

Being rich means not having to wear underwear with holes and stains. Being rich means always using heavy duty aluminum foil. Being rich means using shear bandaids instead of plastic. Being rich means using brand name kleenex and toilet paper. Oh yeah baby, living large.

I crashed my bike a few days ago, ironically while riding home from physical therapy for my bike crash injuries. When you live in a hipster neighborhood, you have to just commit random ironic acts from time to time. I actually crashed because I was riding along on the side of the road and had my eyes on the cars and ran right into a giant pile of sand. Suddenly my bike was stopped, and I kept going, and then I was on the pavement. I bruised my hip pretty badly, but nothing permanent. It made me realize that I've been riding too considerately; I'm too nice to cars, I endanger my own life by riding way over on the right edge of the road to stay out of their way. The edges of the road are a very dangerous place for a bicycle, because that's where all the detritus of the road is scattered. In Seattle it's particularly bad because we have no fucking goddamn street sweeping, so you get things like giant piles of sand on the side of the road.

08-05-09 | Relacy License Notes

The lock-free code I posted with Relacy has a clarification to the license agreement added. If you have downloaded this please read and make sure you are in compliance. I've copied the added text here :

ADDENDUM ON RELACY LICENSE : (revised 9-14-09)

Relacy is now released under the BSD license :

    1 /*  Relacy Race Detector
    2  *  Copyright (c) 2009, Dmitry S. Vyukov
    3  *  All rights reserved.
    4  *  Redistribution and use in source and binary forms, with or without modification,
    5  *  are permitted provided that the following conditions are met:
    6  *    - Redistributions of source code must retain the above copyright notice,
    7  *      this list of conditions and the following disclaimer.
    8  *    - Redistributions in binary form must reproduce the above copyright notice, this list of conditions
    9  *      and the following disclaimer in the documentation and/or other materials provided with the distribution.
   10  *    - The name of the owner may not be used to endorse or promote products derived from this software
   11  *      without specific prior written permission.
   19  */

My work with Relacy is 100% free for any use. However, the original Relacy license still applies to all work product made with Relacy, such as my code above.

The version of Relacy that I built my code with was released under a previous less clear and restrictive license. Dmitry says the new BSD license applies.

08-04-09 | CINIT

Two questions I can't find answers to :

1. Is there a way to tell from a piece of code that you are being called from cinit ? eg. in C++ when a constructor causes some code to run, and that calls some function, and then I get called, is there anything I can check to see that I'm currently in cinit, not main?

(obviously a very evil thing I could do is run a stack trace and see what's at the top of the stack). I can't find anything in C that I can check, because the C stdlib is initialized before me, so to my cinit code it looks just like I'm in the app run.

The reason I want this is mainly for asserting & validation - I want to make sure that my own cinit code isn't calling certain things (such as memory allocation) so I want to put in checks like ASSERT( ! in_cinit() );

2. Is there a way to disallow cinit code in certain modules? For example, having cinit stuff in any library is very unsafe because you have to be wary that your OBJ could get dropped from the link and the cinit stuff will not be run, plus you have order of run issues, it's something I can handle fine in my own projects, but not something I want to force on clients. So I want to make sure that my actual deliverable libraries have no cinit stuff - but I do want cinit stuff in my test apps, so I don't want to just break it entirely. I'd really like a compiler error so I know I did a booboo right when I write it.

08-04-09 | Health Care

Okay, I've been resisting writing about health care, because it's just so depressing and pointless. As usual, the Dems are proposing some very modest reforms that really don't get at the root of the problem at all. And as usual the Reps are going into insane reactionary mode for no reason and attacking them in ways that are just nonsense. The Dem plan does almost nothing to change the way current health plans work, or the way Medicare works, both of which are fucked. The Reps are using this bizarre bullshit that the Dems are going to let old people die - the ironic thing is that if that were actually true I would be way more excited about reform.

We desperately need to do two things that we aren't going to do :

1. Remove doctors from the position of hallowed unquestionable saints. We need more transparency and consumer power. The consumers need to be able to make more of an informed capitalist decision about their own care, and furthermore the health insurance companies need more real tracking of doctor's performance so they can decide who should get paid. There's this false idea floating around that in the ideal system, doctors have unlimited freedom to provide care and the health insurance company gets out of the way. That's retarded. The health insurance company should be your agent to ensure that the insurance pool you're paying into is used well - it should only be spent on people who have real serious health problems, and it should only be used by doctors in ways that are really necessary and cost effective.

I often think about health insurance by comparing it to car insurance. What if your car was wrecked and you took it to a shop and the mechanic just said "trust me; I'm not going to tell you how much I'll charge, and no you can't find out how well I've done in the past" and the insurance company couldn't check to make sure the damage was real and the repairs necessary and fairly priced. Of course that would be a disaster of extortion. The service provider needs to be kept in check by both the client and the insurance company, that's what motivates them to provide good services cheaply.

2. Ration care. Rationing is crucial to controlling costs, and also crucial just for being rational sane policy makers. Of course we all implicitly put prices on lives every day. When you choose to buy a cheaper car that's not as safe, you are putting a price on your life. When the city chooses not to spend the money for separate bike lanes, they are putting a price on life. When you have limited funds, you should allocate them to the place that provides maximum benefit. The correct policy is a kind of rate-distortion analysis slope chasing thing. You set a certain slope - a [Life Value]/[Cost] slope - and any time that slope is better than some threshold, you spend the money, if it's worse than that threshold, you don't. To do otherwise is just retarded.

"Estimates show that about 27% of Medicare's annual $327 billion budget goes to care for patients in their final year of life." - this is a little tricky because it's hard to tell when exactly it is an old person's last year of life. In that final year, 40% of their cost is spent in the last month of their life, presumably because they become in-patients and undergo various emergency life extension attempts. There are tons more statistics like this. The crucial thing is that we're obviously wasting massive amounts of money on these things.

It's important to frame the rationing question the right way; what we're really doing is "health care optimization"; the point is that with cost held constant, we are trying to save more lives. By letting one geezer die, you would have enough money to help ten black people avoid diabetes. And in fact, when you optimize money allocation for the public health, you'd see that actually we'd be better off if a lot of this money we're wasting on health care was spent on more general societal things like safer roads, better standards for bumpers on large trucks, more public parks, more food inspections, etc.

Anyway, my point is not to convince you that optimization is necessary - it's inherently obvious that we should do optimization, and anyone who says otherwise is just a hopeless tard that you can't even debate with. The sad thing is that in our political system it has zero hope of happening. Politicians will say "are you suggesting we should choose to let someone die because they are too old and too sick?" ; of course I'm saying that, it's the only reasonable position to take, and of course hospitals already do it, they just draw the line way too far out; if we seriously want to limit costs we need to draw that line in a bit, and more importantly perhaps just change the entire attitude that the point of the health care system is to spend arbitrarily large amounts of money to make arbitrarily small incremental improvements in life span.

Furthermore, because of the power of doctors and the AMA, #1 is not going to happen either. Doctors don't want patients to know their success rates or their costs, or to see their real alternative treatment options. When you go into a doctor, you should get information like : you could do this or this, for each option it will cost X, and the success rate is Y - but doctors don't want you to have that, they want you to be ignorant, because it lets them charge you too much and treat you poorly without you knowing about it. When you go in with a sprained ankle, right now they send you for an x-ray and maybe an MRI right away. Instead someone needs to say "you could get an MRI, but it costs $10,000 ; or you could just go home and ice it and see if it gets better in a few days".

08-03-09 | How to Reply to Emails

Yet another in the ongoing cbloom.com feature : Teaching You Fucking Tards How to do the Most Basic Life Tasks Correctly.

(I'm not even going to talk about the classic amusing blunders like the "reply-all" when you are trying to reply to just one person on the mail).

When someone sends out a detailed email with multiple points in it, like




You don't just hit "reply" and then randly write a stream of goop. You also don't just reply to only point A or point A & C or whatever. The correct way to reply is :

quote :
[ A ]

Reply to A

[ B ]

Reply to B

[ C ]

Reply to C

Furthermore, if the points A, B & C were very long, you should not quote the entire thing, but rather abreviate them, either using cut and ellipsis, or by extracting just the specific sentences you need to reply to, but you should leave enough context that it's clear what you're replying to.

Finally if there are multiple people on the email, make sure it's clear who you are quoting. A lot of emailers will automatically insert something like :

you wrote :
[ A ]

which is quite useless when there are many people on the mail. You should go in and manually delete the word "you" and change it to the name of the writer :

Douchey McDickWad wrote :
[ A ]

(I'm imagining the NBC "the more you know" thing going by, but instead it says "I shouldn't have to tell you this" or "you fucking tard").

I'm really sick of sending someone an email like : "can you tell me about X? also what's this thing Y? and furthermore Z". And I get a reply that's just like "blah blah Y blah blah Y blah". Uh, hello? What about X and Z ? Fucking compose your email.

In other "people fucking suck" news : my bike has been sitting in the shop done for a few days and they never called me. I finally called in and the person gave me a bunch of attitude, I was like "is my bike done?" and they were like "well, did we call you?", and then lo and behold it was done and they were like "oh".

In "people are rad" news, I went to 2020 Cycles (I was avoiding it because it seems ever so precious and hipster, and because Sharp said they did bad to his bike (if you've eaten recently, don't follow that link, because it's the most nauseating hipster bullshit you will ever see; oh yeah, we're a bike shop, but we have indie music shows and we cultivate community and have local artists and fucking vegan coop hand-knit bicycle seats and we all wear american apparel and grease your ball bearings with the grime from our bodies! yeah!) ); anyway, they pulled my bottom bracket while I waited, and charged me only $5 for it, and they were super nice and helpful and friendly and knowledgeable. Fucking Recycled Cycles charges $20 for every little service thing, no matter how trivial. (a fucking bottom bracket lockring spanner is yet another obscure tool I don't have; hopefully this is the last of those problems. Every time I try to do anything crafty, be it woodworking or working on my car or bike, I always run into instructions like "now use your rotary mitering lathe bit with the quadrangle star attachment on your half wilsonized wall-mount jig saw" and I'm like WTF I don't have that shit this is ridiculous).

Also : the new Surly Gourmand is a riot.

I’m sure some people really like Perche’ No, but those people are probably retarded.

08-02-09 | Private Beaches

Seattle's got a number of private beaches which are just begging me to sneak into them. The whole idea of the private beach seems very elitist to me after living in CA where there's a wonderful law making all beaches public property. It makes me picture the European Riviera where you have to be a member of a beach club and buy a chair or something to get beach access, and you must be able to point your nose upward to get in.

One is Windermere Park on Lake Wash. It's got a pretty serious gate at Ambleside which you can see from street view . Apparently they lock the gate at 7 PM each night and check membership during the day. That gate definitely looks hoppable, but another entry option would be to kayak up from UW. Windermere Park is labelled on Google Maps which is a bit misleading as it's not a public park.

Perhaps even more appealing is the Blue Ridge Private Beach . I guess Blue Ridge used to be a private community before it was annexed by Seattle, and it still has some of that left over; there are still homeowner rules and membership fees. The park is not labelled on maps, but you can clearly see the path to the beach in satellite view. The fencing around the park looks like pretty serious barbed wire, so the best entry option seems to be walking the train tracks up from Golden Gardens or down from Carkeek.

BTW, I've never understood how people die on train tracks or why there are so many signs about how "dangerous" it is. All around Carkeek there are these signs to stay off the tracks and that the trains are "fast and quiet". Uhh... I'm pretty sure if one of our trains surprises you, you must be absolutely drunk out of your mind, like almost unconscious. I mean, I understand with like a bullet train, those are in fact fast and quiet (though they also run on very straight tracks, so you can see them coming for miles), but our trains go maybe 60 mph and rattle and screech like motherfuckers. You could be a deaf giant sloth and still have no trouble avoiding a train. The signs should really say "warning : don't get black-out drunk on train tracks". Or the sign should just have a handgun on a chain and say "please kill yourself here so you don't derail the train".

08-02-09 | Books

Some recent book reviews :

"The Thought Gang" - Tibor Fischer. Absolute drivel. Not one redeeming thing about it. Tries so hard to be clever but there's not one interesting thought in the entire thing. The Z's and made up words and obscure vocabulary are ever so precious and pretentious and tedious. The side characters (mainly Hubert) are almost lively and wacky enough to get it back onto the rails as a Clouseau-esque madcap crime romp, but the boring Eddie and his awkward forced attempts at philosophizing jolt you back into the shitty worthlessness that is this book.

"My Life in France" - Julia Child. This was pretty mediocre. It was written "with" (aka "by") a real author, but apparently he's also a terrible writer, because the prose is very stiff and unpleasant. There are obviously parts where it's trying to get you excited and convey the wonderful food epiphany that Julia had in France, but it doesn't come across in the awkard writing. The thing I really appreciated was the feel I got for Julia's energy and character. She's extremely awkward, and kind of a bitch, but she's full of excitement and very welcoming to the people she loves. The thing I found really inspiring was the way she just casts aside people who would hold her back. They move to France and the other Americans working with Paul are boring drips, so she has nothing to do with them; one of her co-authors isn't pulling her weight, so she gets cut. Over and over there's a sort of unapologetic selfishness that I find very refreshing and admirable; it's not mean spirited or sour at all, she has great enthusiasm for the things she loves, and if you're not making things better, you're out, tant pis. If you want a memoir about the golden age of the foodie revolution, read Jacques Pepin's "The Apprentice" instead.

"Travels with my Aunt" - Graham Greene. Meh; this didn't really do anything for me. It's sort of obvious exactly what's going to happen from the very beginning; obviously the boring stick in the mud guy is going to get taken around by his aunt and discover his love for excitement and transform, blah blah. It's got a bit of a "Harold and Maude" vibe to it (which was also just awful BTW) in that "Maude" is just this absurd character who seems to have no concept of laws or responsibility and seems to magically skate through life unharmed. The book also is just terribly constructed, random bits will be incredibly drawn out, and then huge things will happen suddenly in one paragraph.

Some douche recommended Decca Aitkenhead's "The Promised Land" to me; it's just awful. First of all she's the worst kind of clubber (or just generally the worst kind of person) ; she wants to spend all her time talking about how great *her* group is, the fact that it was always better in the "old days", that it's not good now that it's sold out and gone mainstream, she's got this absurd hypocritical snobbery; it's actually quite hilarious when she describes her group's type of ridiculous rave dancing as if it's so cool, and then in her travel writing she mockingly pokes fun at other group's style of foolish rave dancing. You fucking twat, the whole wonderful thing about rave dancing is that it is always ridiculous, and everybody knows it's fucking ridiculous but they don't care because they're high and the good people relax their need to look cool and be self conscious.

There's a blurb on the cover of "The Promised Land" comparing it to Bill Bryson. They mean that to be a positive thing, but in fact it's a warning : they're both fucking annoying cunts. It reminds me a lot of "A Walk in the Woods" actually - in both books the author does the experience all wrong, doesn't prepare, doesn't research, and then just whines and complains and writes snarky condescending annoying uneducated drivel that's supposed to be funny but is just sour and sad for them. They both completely fail in what I want from a travel book, which is to be inspired and excited and to read about people who really dive into the experience and do it more immersively than I could myself. There's a whole category of modern travel writing where the author acts all superior to what they're observing and sort of bored and complain about the annoyance of travel. God, if I wanted that I would just listen to myself! I don't understand why you retards keep buying these books, and I really wish you would stop recommending them to me. Both Decca and Bill manage to take something that I really love and make me feel sour and unhappy about it and make me less enthusiastic about leaving my house (partly because I might encounter a fucking cunt like Decca or Bill).

I don't find condescending superior snarky writing to be at all funny any more. Maybe that's because I've been doing it myself for so long and I fucking despise myself, so what I want in a book is the opposite, I want an author who's energetic and accepting and positive and open minded and jumps right in to other people's customs and tries to see what's good about them.

08-01-09 | Eggs

Two techniques for cooking eggs :

Oeufs au Miroir : this is just the best way to make a fried egg with a soft yolk. It's far easier to control and hit the perfect doneness level than with the traditional technique, and the white is more set. (with a normal fried egg you have the dilemma that you want the yolk just barely cooked, but undercooked white is disgusting, so you have to flip the egg just briefly and try not to break the yolk and be careful not to overcook it).

The technique is very simple : heat a nonstick pan to medium high as you would for a fried egg. Put butter in pan and let it get very hot but not browning. Gently crack in eggs. Immediately put a lid on the pan and turn it down to medium - medium low. The carryover heat in the pan should brown the bottom of the eggs well, you should hear it crackling. Do not take off the lid if at all possible - you need the steam in the enclosure to stay trapped inside. You want to judge doneness with your ears. When the crackling slows and the eggs become quiet, they're probably done. The top of the egg should be glassy and smooth (hence the "miroir") the yolk should be just starting to gel like a creme anglaise. If you like you can flip the egg over when serving so it looks like a fried egg (one side is fried, one side is steamed).

The traditional French technique for Oeufs au Miroir calls for adding a few drops of water to the pan when you crack them in, to aid in the steaming. I find that our shitty over-large eggs are so full of water that this isn't necessary and don't usually bother.

Scrambled eggs : most Americans make disgusting overcooked rubbery scrambled eggs; the traditional French technique for scrambled eggs calls for cooking them very low and slow with added dairy, which makes them almost a soup, which I also find disgusting. I think the sweet spot is somewhere in the middle, I want them cooked through, no raw or runny bits, but just barely set.

The hard thing about cooking eggs is that they cook very fast, so they quickly go from "just right" to "overcooked". This is why so many professionals are terrified of cooking eggs. Part of the difficulty is that if you plate them hot, they will carryover so even if they were perfect in the pan, they are overcooked when you eat them.

I actually think both of these things can be addressed pretty easily with the application of reason and logic. The way you solve the difficulty of eggs flying so fast past the target doneness is just to cook them slower. Similarly the way you solve the difficulty of carryover is just to plate them at closer to room temperature. Hence we have a very slight modification to traditional scrambled egg technique that I think makes it much easier to do well :

Heat a pan to medium high and butter, just like for a fried egg. Crack eggs into pan, they will be crackling and cooking very fast at this point. Immediately turn the pan *OFF*. We are going to scramble them entirely with the residual heat in the pan and let it cool down. Stir the eggs a bit gently, the pan should still be very hot at this point and cooking fast, but also cooling fast. You can control the amount of cooking a bit by the amount of stirring - more stirring = less cooking. The eggs will be done in about a minute; as they get close to done the pan should be coming down to lukewarm, so you don't have so much risk of flying past doneness. Plate and again stir vigorously so they don't trap too much heat as they sit on the plate.

Breakfast scrambled eggs I like just plain. For dinner scrambled eggs, add a tiny dash of cream and a tablespoon of grated parm part way through cooking. Parmesan is an operator which converts eggs into dinner. Top with chives or parsley.

07-30-09 | Night Swimming

The Seattle Heat wave of 2009 is over. It was dog-shaving heat. It was old-people-killing heat. It was delightful, I'm kind of sad. I hadn't yet gotten to do two things I really wanted to do during the heat wave : 1. swim in the Sound - when it's 100 degrees out is the only time it's sactually tolerable to swim in the ocean here, and 2. skinny dip in a lake at night.

I did go swimming in Lake Wash both of the last two nights. Last night when it was 101 here it was a crazy scene, cars were parked all over the sidewalk around the lake because there was nowhere to park. Of course the fucking dickweed cops were out writing parking tickets and alcohol tickets; can't you make an exception for the hottest night ever in Seattle? Tonight it was still warm but not so severe and less of a scene.

It's incredible being out in the lake during sunset. If you swim out a ways, you feel all alone, and you're wrapped in nothing but sky and water. The sky turns orange and pink, and it reflects off the lake in glittery flashes of specular as the chop hits just the right angles. There are incredible views of mountains all around that become black silhouettes. There's a strange way that being out in the lake actually makes me feel closer to all the distant views. Being in the water, you're tossed around by the swells, you're physically connected to the world outside your body. Especially with the water being so warm, it's almost like being in the womb. It breaks down that barrier, that separation that we feel every day from all the things that surround us, like we're a miniature pilot inside our skull and even though our body's hand is touching something, the miniature pilot is still separate from it.

After the sunset, the moon was bright in the sky, and the lights along I-90 stretch out like a string of white christmas lights and reflect in the water. I floated and swam around for a long time.

Back on shore, people smoked and drank and spilled beer and peed in the bushes and grunted and beat each other with sticks and flung poo at each other.

07-31-09 | Visual Assist X is fucking annoying

In other news, I'm trying to get on VS 2005 from 2003. I'd like to keep my 2003 install working, but fucking Visual Assist X stomps on my Visual Assist .NET ; the whole reason I've stuck to 2003 all this time is because Visual Assist X sucks donkey balls and VA.NET is the bomb. Fuck. I wanted to keep my VS.2003 + VA.NET because it's my favorite editing environment, I would still use it to write my old personal code, but if I need 2005 for work then I need VAX and that kills 2003.

If you read the VA.X forums, it's obvious they have the same dick-wad attitude as perforce. Tons of people complain about the same issues, and all the replies from VA.X people are like "is there actually a problem with how it works now" or "try it for a while and see if you get used to it". FUCK YOU I want it the way *I* want it, not the way *you* want it.

My main complaint with VA.X is with the whole listbox/suggest system. It pops up all kinds of flashing boxes all over the place which change and move around as you type. It's epilepsy inducing just like a Dragon Ball Z episode. It's like a spastic retard with tourettes yelling at you as you type. You type "pr" and it starts yelling "printf! no process! procedure!". My solution at the moment is just to turn it off (don't ask me how I turned it off; I couldn't find an option to turn it off and was just randomly clicking things and then it was off; I'm not sure I could turn it back on if I wanted to). So all I'm getting basically is enhanced syntax coloring.

What I would like is for it to act like VA.NET - first of all it needs a typing delay so it doesn't pop up unless you pause for a second, and then it should be slim and subtly colored and stable. And I really don't need a list box of suggestions, I only need the one most likely, if it's not right I'll keep typing.

If all you want is "open opposite" and "open project file", NiftySolution is the way to go. For a little bit more, WorkspaceWhiz also seems pretty nice and lightweight, though it's currently broken for me, I'm in touch with the dev and hopefully it will get fixed.

One nice thing that WorkspaceWhiz and Visual Assist both give you is browsing without browse info. That is, they do their own parse of the text directly and don't rely on the MSVC browse info build system which is horribly broken (one of the many problems with it is that you get no browse info on files until you compile them).

NiftyPerforce for VC2005 + is totally superior to the Perforce provided plugin. Good job.

It sort of annoys me that I can't write code without an editor enhancement any more. But I guess that's like saying I'm annoyed that I can't commute without a car.

07-30-09 | cbloom.com

I just had a look at the cbloom.com stats for the first time in over a year, and it was quite a shock!!

A while ago I tried putting up a wordpress blog; I never really even got it off the ground, but apparently it's the most popular page on my site :

requests    page
24832       /wordpress/
15681       /wordpress/wp-comments-post.php
12585       /rants.html
10966       /robots.txt

And lots of bots are trying to make comments. If you look at the browsers used to visit the site it's obvious why :

reqs    pages   browser
8276    8082    Baiduspider (http://www.baidu.com/search/spider.htm)
6415    4186    Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; )
4299    3636    T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240)
5062    3102    msnbot/2.0b (http://search.msn.com/msnbot.htm)
4420    3001    Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
5902    2903    msnbot/1.1  (http://search.msn.com/msnbot.htm)
2700    2700    Mozilla/5.0 (Macintosh; Mac OS X AppleWebKit KHTML, like Gecko, Safari/528.16 OmniWeb/v622.8.0)
3236    2212    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
3081    2057    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)

Baiduspider I guess is a chinese index bot, so I'm not sure exactly who's trying to autopost to my wordpress. #2 is the Yahoo search spider. The "T-Mobile Dash Mozilla" apparently is also a search bot for MSN Mobile. Then we get three more search bots, and then we get Safari ! It's not until the #8th most used browser that we see just a normal Windows Firefox (correction : actually Mozilla/4.0 means IE).

Not surprisingly, the Googlebot seems to be way more polite about not abusing your site. Even though it indexes my site as well as anyone, it hits me far less. (MSN is actually the worst by far if you add them up). (of course some of these could be spam bots pretending to be search bots).

The other funny thing was the referring search words - the top nine are all poker related, it's not until #10 that you get "compression".

In semi-unrelated news, god damn all you ghetto fucking developers and your ghetto ass command line parsing skills. Any time I see a message like this I want to punch you in the nuts :

"Unknown argument -? ; use --help for help"

URG WTF ; 1. just go ahead and show me the help if my args are no good, and 2. how fucking hard is it to recognize a question mark? Just add it to your switch cases :

case 'h':
case '?':
    // show help

How hard was that?

The one that bugs me more is not recognizing switches unless they come in the right order, like I was just using "NcFTP" today and it says

Usage: ncftp [flags] < host >

So of course I type

ncftp www.cbloom.com -u xxx

And it's completely confused. WTF, how hard is that to handle ? (voice of Gob) COME ON!!

(BTW NcFTP is a handly light FTP client that supports recursive delete get and put).

The standard way I use command line apps is to write some command line, up arrow to edit it and tack more flags on the back. You should always handle flags occuring at any point of the command line.

Another one that bugs is people who expect either a space or not space for the payload of a flag ; eg they require either "-s 1" or "-s1" but don't support the other. The big problem is it's not standard so my hands don't have a reflexive habit of which way to do it. Of course you should just support both, it's pretty fucking trivial.

In cblib I use this :

int argint(int argc,char * argv[],int & i)
    char * cur = argv[i]+2;
    cur = skipwhitespace(cur);
    if ( *cur )
        char * endptr;
        int ret = strtol(cur,&endptr,10);
        if ( endptr != cur )
            return ret;

    if ( i == argc-1 )
        FAIL("no int value found for arg!");

    return atoi(argv[i]);

A standard cbloom arg parser looks like this :

int main(int argc,char *argv[])
    lprintf("newdct built %s %s\n",__DATE__,__TIME__);
    lprintf("usage : newdct [options] < from > [to]\n");
    lprintf("usage : newdct -h for help\n");

    int killBands = 0;
    const char * fmName = NULL;
    cosnt char * toName = NULL;

    for(int argi=1; argi < argc ; argi++ )
        if ( argv[argi][0] == '-' )
            switch( toupper(argv[argi][1]) )
            case '?':
            case 'H':
                fmName = NULL;
                argi = argc;

            case 'K':
                // handles -k N and -kN and "-k N"
                killBands = argint(argc,argv,argi);
                lprintf("got option : killBands = %d\n",killBands);
                lprintf("warning : bad option ignored : %s\n",argv[argi]);
            if( fmName == NULL ) { fmName = argv[argi]; }
            else if ( toName == NULL ) { toName = argv[argi]; }
                lprintf("warning : ignored extra arg : %s\n",argv[argi]);   
    // toName is optional
    if ( fmName == NULL )
        lprintf("HELP : \n");
        // ...

Do it my way god damn you.

07-28-09 | Brunch

People are obsessed with going out for fucking brunch. It's like they think it's some kind of fucking magical treat. Do you know you can make pancakes and french toast at home any time you want and it's really fucking easy? And like mimomas is just fucking orange juice and champagne, it's not like you need the damn brunch fairy to cast a spell.

XX always wanted to go to brunch and I almost always refused because I'm a dick. After that I decided I should be a better boyfriend and try to do the things that my girl wants to do even if I hate them, and try to have a good attitude about it and enjoy it, not just go through the motions and be a whiney prick like I usually am. So with YY I went to brunch a few times and tried really to be a good sport, but my god it sucks.

Reasons why I despise going out for brunch :

1. By the time I get there I'm going hypoglycemic and I'm getting cranky from having no breakfast; if I was fed right away that would be okay, but it's always a huge wait, and of course the whole point is to take it slow. So I guess I have to eat breakfast first, but that sort of spoils the brunch eating.

2. It's always slammed. Because every tard and his mother all love fucking brunch it seems there are never enough places that serve brunch and they're jam packed. I don't even mind the long wait that much, the really annoying thing is that people are hovering around your table because the restaurants are packed and waiting people are getting crammed into the diners, the waitress is in the weeds and running around throwing syrups at tables, and the kitchen is just rushing things. It all makes me very antsy and uncomfortable. I like to take my morning slow and very relaxed, have coffee, read the paper, greet the sun, no stress in the morning.

3. I can't eat the pancakes or french toast or whatever because they're fucking huge sugar bombs and I'll feel awful for the rest of the day. But they tempt me because they are the only good things on the menu usually; I get stuck ordering the eggs and sausage or something and it sucks. Restaurants in general cannot cook eggs worth a damn; the eggs are generally cheap, rubbery, tasteless, overcooked and oily. I usually wind up ordering poached eggs because they at least won't be fucked up too badly.

4. The coffee is disgusting. Even the fancier places that do espresso or french press usually serve vile swill.

5. I have to change out of my pajamas, which totally spoils my relaxed morning vibe. Once the pajamas are off I want to get out and do something, not sit around for hours having god damn brunch.

The Tom Collins is the drink that I always am wanting but never think of. On a hot day when you want something crisp and slightly bitter and slightly sweet, it's perfect. Often I'll get a greyhound or a gin-tonic or vodka-soda or something, but those are all too bitter/sour or too tasteless; the Collins is exactly what all those drinks wish they were.

I also approve of the Manhattan, the Old Fashioned, the Sazerac, the Caipirinha, the Mint Julep, the Mojito and the Daiquiri. I do not approve of anything involving a red or blue colored liquor. In Seattle there's a craze right now for Elder Flower liqeur which is fucking vile. Alcohol should not taste like perfume.

How do I make Firefox never ever never play a sound from any source ? No flash playing sounds, no embedded quicktime or any other players. I don't ever want to hear a web page. (I wish I could browse the web without ever seeing fucking Flash again, but then I also wish people were nice to each other and sex was casual and easily available; it ain't gonna happen) BTW for example, the other day I was on this site Tree Fort Bikes which is actually one of the best flash sites I've ever seen; it's fast, has good features, has a well designed UI, and I started thinking "you know this flash site isn't so bad" - but then I tried to drag a link to an item out to my desktop - OH NO! And then I tried to middle click an item to open it in a tab - OH NO! Fucking flash.

07-25-09 | Blurg

Went for a great ride this morning, the legs are coming back around. I think my body's getting better, but I have had a few scary incidents recently where something clicked in my neck and my whole head went numb. That's never happened to me before. It happened for the first time two days ago at work. I was at my standing work station and just kind of rolling my neck around to stretch it a bit, when suddenly something popped and my ears and the back of my skull got all warm and tingly feeling and went numb. Very scary. Then just today after biking I got a similar thing. Biking is really not good for the back and neck but fuck it, I love it, if it paralyzes me, I'll just kill myself and it's no big loss. In the mean time do what you love.

Anyway, feeling great, I went to check out Carkeek park. It's kind of a ghetto park and the tiny beach gets very crowded on a nice day, but the view across the sound to the Olympic Mountains is pretty great. It was hazy today, but I bet on a clear day the sunset is spectacular (Spanish style : spectacularrrrr , or Pirate style : spectacul-yarr). Anyway, on the beach there was this absurdly hot chick hoola hooping in a bikini; that's just cruel, it was that slow sensual hoola hooping, like she was giving the air a lap dance. All around were cute little kids playing and laughing. It all made me really depressed. Leaving my house is almost always a bad idea. Seeing people who have what I want but cannot get is so depressing. It's kind of like watching documentaries about starving africans and genital mutilation and shit like that - yes, I know that all happens and it's fucking awful, but I really don't need to sit and watch a big show about it, it's just going to make me depressed. Some of my more nerdy friends seem to have found a path to peace in life by just not going outside. I mean they go outside in a literal sense, but the outside world with its fun and children and sex and laughter is just completely across a line that they don't cross or even think of crossing; that's sort of horrible, but it also simplifies things a lot. Certainly if you know there's something that you can never have, it's foolish to pine over it endlessly, just move on and make your life about something you can have.

There's this Norman Mailer quote that I put on my own blog long ago, something along the lines of "I achieved happiness in my old age because I stopped trying to be what I'm not". That true, but it's also trite. The hard thing is not realizing that fighting against things you cannot be will make you unhappy - it's about deciding which specific things are things you can never have (and thus should just give up on and cut out of your life). and which are things that are hard for you but worth fighting for (in which case giving up is cowardly and self-destructive).

I got my new bike in the mail :

new bike

I stripped every component off it so it's naked. I'm waiting for all my replacement mail order components to show up now. It occurred to me the other day that I am literally having a bicycle shipped to me in the mail one piece at a time. It's quite ridiculous. Sadly, when I took apart the headset I found the races were badly pitted. Changing the races is one of the few jobs you can't really do at home, so the bike is now in the shop getting a new headset. (for example, here's an really amazing page on how to change a headset , and here at Dave Moulton's great blog is a crazy page on how to change your headset at home - that guy is fucking nuts, just take it to a shop). To appease my sudden urge to work on bikes, I overhauled my Litespeed, took it all apart, washed it with soap and degreaser, then put it back together and regreased everything. It was fun and it's clean as a whistle now and runs smooth and quiet; a lot of the little hitches and noises on an old bike are just caused by crud stuck in the drivetrain. I'm worried that my sudden burt of enthusiasm for bike building may wane by the time I have everything I need. It's very important for me to be able to pursue ideas immediately when I have the energy for them, and to not have too many different balls in the air at the same time.

In other news : vegetarians and dogs are fucking killing me. The type of girl I like is liberal, into food and nature and gardening and hippie shit like that, the outdoors, hiking, camping. Well, that portion of the population also happens to be dominated by fucking vegetarians and fucking dog owners. Vegetarians don't bother me if they're not in my life, but I can't date one; food and cooking is just way too important to me. And dogs - I don't understand why people think it's okay to let filthy animals into their homes. They lick you and get that stinky slobber all over you (the dogs, not the girls). I'm perfectly happy to play with a dog at a park when I'm getting dirty anyway, but I don't want one in my home, and I'm literally terrified to sit on the furniture in a dog owner's house. Blurg. Oh, and a lot of people smoke up here; one of the many nice things about SF is that basically nobody smokes so you don't even have to worry about whether someone you meet is a smoker or not; in fact I completely forgot that it was an issue. And huge gross tattoos and piercings are very common up here too. Double blurg.

Also, I'm infuriated by relationship advice from people who married their high school sweetheart or someone they met in college. Good for you, I'm happy for you, you got lucky early on, but you don't know the first god damn thing about adult dating or meeting people, so just shut the fuck up. One of the worst pieces of advice is "just do the things you love and meet someone naturally that way". Retarded bullshit. The things I love are pretty much all solitary, since I fucking hate people. The actual good advice is something more like "do something social involving others that is sort of vaguely related to one of your interests that you think only people similar to you would do". So like I guess I could go on some group bike rides even though I fucking hate group riding, or I could go do some fitness classes or something even though I fucking hate fitness classes. Blurg.

It's 80 degrees here now and my laptop fan is really angry. It's running at top speed all the time. Next week it's going to be 90 here. I might have to buy an air conditioner, or just stay off the computer, or it's gonna melt. The other day my laptop turned itself on inside my laptop bag; I guess it opened a crack or something; when I got home from work and unzipped the bag I found the fans running in overdrive in vain as the bag is quite thickly padded and sealed; the poor lappy was scalding hot to the touch. Definitely a concern. Personally I fucking despise the fact that lappies tie their power state to the screen being open or closed; on every lappy I've ever seen, that switch becomes flaky. I would way rather have a big hard switch for on/off (really asleep/awake).

ADDENDUM : I caved and bought an air conditioner. In a way I like that my apartment basically has no climate control - no ac, terrible heat, windows that are hardly there - it forces me to really experience the seasons; I was freezing in winter and had to get under the blankets and cuddle with a warm body and bake lots of gratins; now in the summer I have to get naked and spray water mist around the room and get outside to the park to try to get some breeze. In another more practical way, it sucks balls.

07-23-09 | Intolerance

The ultra-liberal idea that we should respect and value everyone and their different beliefs is a load of pussy bullshit. All the condescending more-accepting-than-thou Yoga / New-Age people are really very unaccepting; I want to be impatient, I want to be full of fire and anger, I want to be violent, and they are very rude and intolerant and non-accepting of that. In reality they are accepting of you only if you subscribe to their value system (peace and kindness and whatnot). I think the whole idea of "respecting other people's beliefs" is a load of shit. I have great respect for their *right* to believe a load of crap, but I don't actually respect their beliefs. I think yoginis and christians and muslims and the whole lot of you are fucking crackpots and I totally do not respect your life choices, but I would fight with my life for your right to do what I believe is silly and wrong (EDIT : no I wouldn't; in fact I wouldn't lift a finger if freedoms were denied that didn't affect me personally). I ask that you also respect my right to roll my eyes at you and huff in disgust at your foolishness.

I want to do Yoga but I really can't stand all the "breathe in the energy and feel your pancreas release" nonsense.

Amusingly, the ultra-right-wing Republicans are almost identical to the ultra-right-wing Liberals. When Rush Limbaugh or Toby Keith talk about how we should fight for our "freedom", what they really mean by "freedom" is the right to do things that fit in their own value system. They want the land of the free, where "free" means free to be a good heterosexual christian consumerist semi-racist suburbanite.

It occurred to me that the antipathy between cars and bikes is a lot of the same thing. In both directions, it's a form of bigotry; it's a hate for another group that you stereotype and don't relate to. I hear a lot of people say these things all the time jokingly basically "ha ha I don't like this thing so let me fuck it up" ; it's only being considerate of how your actions affect other people in realms that you personally value. You see this constantly, the guy driving slow in the fast lane because "nobody should want to go faster than I'm going", or the guy who litters cuz he doesn't care about seing litter, etc.

Part of the reason why the streets are so bad for bikes & pedestrians is because democracy fucking blows and the tyranny of the majority. If you take any given street that you want to improve - take for example the 1st Ave / Pike St around the market - that street should obviously be closed to cars and have nice bike lanes and whatnot. But the people who live there and go there are a small minority of the total population - there's a whole mess of drivers who live in the sticks that never go there and all they see is it might remotely badly affect them.

It's sort of odd that Seattle seems to have a shit ton of cyclists, because it's really a horrible horrible place to ride. The weather sucks 8 months out of the year, the roads are full of pot holes, there aren't good shoulders or bike lanes, you can't get out to country roads easily at all, and even when you do they suck. It's just awful in almost every way. And yet there are tons of them.

I'm exhausted today, I really need to get to sleep early, and I'm just dreading the noise the neighbors will be making tonight. I feel like every night when I'm just begging for some sleep, it makes them have a noisy party. I know that this is my own memory-perception-bias ; nights when I'm exhausted but they don't have a party I don't remember, the particularly bad events have a much stronger imprint in my memory so I overweight them in my perceived random sampling. Now, of course I am aware of this, so I can try to correct for it and downweight to reproduce a true sampling, but it's very very hard to correct for you biases. I'm aware of this with all kinds of things, like if you're talking to someone very charismatic and his ideas all sound very good, I'll think to myself "hmm I need to downweight my perception of these ideas to compensate for the charisma factor", but it's almost impossible to find the right amount of compensation.

Girls are very attracted to dreamers and visionaries and people who talk about completely unrealistic utopian ideals, like making the whole city of Seattle pedestrian-only or getting the US on 100% renewable fuels, etc. It's not interesting, it's juvenile. I used to be amused by those inspiring utopian fantasies, but now I have zero tolerance for them. As soon as someone starts talking about something that will absolutely never happen I lose all interest. It's far more interesting to talk about how we might actually improve things in the real world.

Just the mathematics of being single are depressing. You have to flirt with ten girls to get a date with one. You have to go on ten dates to find someone you want to have something long term with. You have to have ten long term relationships to find the one that works. It's unbearable.

Fucking Google Reader. There's no way to jump to the selection of a given blog from the post in the "new items" feed. For example, today Wouter posted this entry "Freedom of Information" where he says he won't be putting out his ideas for free. In anger, I want to unsubscribe from his blog feed. I CAN'T FUCKING DO IT FROM THE NEW ITEMS AREA. I have to go find that blog in the list of subscriptions, click on it by hand to select it, then unsub. Stupid.

I'll do you like a truck is awesome. I think it must be a joke (?). I found it by randomly watching techno videos with hot chicks in them, which is a rather amusing way to pass the time. (The song Day 'n Night has been by guilty pleasure for a little while now).

Roads NW is a pretty awesome sight for nice roads roads to drive in the PNW ; it's aimed at motos but I can abuse it to find driving or biking roads.

07-19-09 | Summer !

Peaches and tomatos are here at the farmer's markets; hallelujah. There's this stand at the Broadway farmer's market (usually set up right near the corner of the L) that's a family farm, and the stand is staffed by the farmer's three teenage daughters. They're all crazy skinny and have ginormous boobs, like cantaloupes; it's actually kind of freakish, it's almost so inhuman as to be unattractive (but not quite). It makes me imagine rolling in the hay in the barn, the farmer with a shotgun, and the Seinfeld where Newman flees into the corn. Also, quite fittingly, they sell the most dependably ripe juicy peaches I've ever found. It's funny, those hoity-toity stands that have big signs saying "don't squeeze the peaches" and have each peach individually supported in its own plastic cup - those tend to suck. This place has the peaches just all tossed together in a box and they're soft and fresh picked and super wet and sweet - and cheap. They also have some superb sweet-tart cherries that are great for pies or preserves or I even like them just to eat.

I'm sort of dismissive of fancy tomato or peach varietals because the dominating factor by far is ripeness and freshness. It really doesn't matter if you have a Mackinaw Peach or a fucking Amish Cherokee Brandypaste Tomato, if they weren't picked that morning at the peak of ripeness, they're fucking garbage. I'll take an Early Girl tomato and a standard yellow peach that are optimally ripe and soft.

Yesterday I biked down to Green Lake, had a swim and biked back. It was quite delightful, not too far, the bike back up to cap hill is a little tough but doable (I wish the hill was the opposite way though, it would be so much sweeter to bike up hill and get to jump in the lake to cool off and then have a downhill to get home). Ravenna Ave is really nice to bike on, it has a big bike lane, and it's a wide European-style "grand boulevard" with trees all down the middle. The Eastlake Ave area under I5 is pretty neat, you can see the ship channel, and the underside of the I5 bridge is cool. It was here that I saw something unbelievable :

First, for background, Seattle drivers and pedestrians are all ridiculously slow and passive and retarded. For example there are a ton of people who think they are "nice" at stop signs. I'll pull up to a 4 way stop, and there's someone there who got there like an hour before me (way before me), and I pull up and they don't go; I glare at them, they wave at me to go. YARG BLURG don't fucking wave at me it's your turn, just fucking go! You're not being nice, you're being a dumb dick. The pedestrians here are super passive and people inexplicably don't J-walk. It's so bizarre and disconcerting coming from anywhere else in the world, I'll see pedestrians just standing at a corner, with not a single car anywhere in sight, and they just stand there and wait for the light to change. WTF. It's especially bizarre like late at night and such when the streets are deserted and people still stand and wait at the corner.

Ok, so I'm going over Eastlake Ave to cross the bridge over the shipping channel and the drawbridge is up to let some big boats through. So I pull my bike over into some shade and get off it to wait. It's taking a long time, I'm just chilling. Now, the traffic light right in front of the bridge is of course red and will stay red the entire time the drawbridge is up. This at Fuhrman Ave E. Then I notice, there's a girl standing at the corner on the bridge side of the street waiting to cross. I look at her; I'm kind of confused. I continue to sit and wait. She stands there at the crosswalk waiting on the bridge side of the street. The light is still red - it won't go green until the bridge goes down. She's just staring at the walk/don't walk sign, which continues to say "don't walk". Note there is absolutely zero cross traffic, there is a big line of cars building up on Eastlake waiting for the bridge. Finally - 5 or 10 minutes later the bridge comes down and the light changes and then she crosses. WHOAH ZOMG. My mind is continually blown here. I think it would have been appropriate if the cop who gave me a ticket for proper merging to show up and give her a ticket for not j-walking.

In other news : fucking hell, cap hill needs a casual tapas place or a cafe with a nice patio or something. The evenings are so long and lovely here and I'd like to go sit somewhere pleasant and have some tasty savory snacks and delicious drinks, but there's just nothing.

07-18-09 | Alcohol is Poison

I quit drinking about two months ago; I've had a few sips out at restaurants but basically haven't had a drop at home. I just decided to stop for a while and see how it went. It's amazing, it's better than I ever imagined; my body feels energetic and healthy all the time. I sleep better, have more hours of the day when I'm thinking clearly, am generally happier. I guess that's all obvious, but the amazing clear feeling in my head all the time was unexpected.

There have also been some surprising benefits. I'm flossing for the first time in my life. I'm actually doing dishes every day (which would amaze people who have seen my past dish-doing habits). These basically come from the fact that I now have the long horribly long boring empty night time to fill with random activity.

One thing I've always noticed is the massive gulf in productive life activity between nerdy ADD people who don't drink and "normal" people ; you ask the nerd what he's been up to last week and it's "learned to play the harpsichord and composed a sonata, created a board game and had the pieces CNC cut, built my own submarine" , you ask the normal person what they've been up to "oh, you know, hanging out, having some beers". Mmmm, good for you.

The other big thing affecting me is that it's so much easier to be fit and healthy and happy when you're single. I always get out of shape when I get into a serious relationship. It's not that I got my goal and don't want to work out anymore, it's just that there's no time. You spend all your free time hanging out, and you do a lot of things in relationships like drinking and eating out and such that really mess up the body. When I'm single I can do weird stuff like just eat at all kinds of odd hours when my body tells me it needs food, or eat around a workout. Like I might intentionally eat dinner at 5 PM so that when I got off work at 7:30 I'm all digested and can work out then. The whole "having dinner with your mate" thing is really tough for fitness.

If you're trying to rehab an injury or make a big shift in your fitness, I highly recommend cutting out alcohol. It dehydrates muscles, decreases anabolism, hurts your sleep. It's also just a lot of empty calories, so it's an easy way to decrease your caloric intake without hurting your nutrition. It also leads to a lot of secondary bad behavior, such as over-eating, late night eating, etc.

Random : The Firefox spell corrector wants me to spell "judgement" as "judgment" . WTF ? Apparently "judgement" is British English and "judgment" is American English. "judgment" is so wrong ; that "dgm" is so ugly.

The coppa from Armandino's Salumi is fucking fantabulous. (thanks!)

I biked the Juanita hill the other day; it's not a bad climb actually, it's a good mild difficulty overall, but nowhere too steep for my double. The only shitty thing about it is all the fast traffic, and there are a lot of big branches in the shoulder from all the trees around, so it's very dangerous to take the descent fast. Some commuter guy just destroyed me up the climb; I stuck to him for about a half mile, but I can't compete with someone who does it every day.

Also, if you want to see an indictment of modern American society, just go to Kirkland on a summer Friday night. It's a bizarre confluence of all the worst dregs of America - there are hordes of cackling suburban soccer moms, spoiled 16 year old girls trying to park their Range Rover head first in tiny parking spots, frat boys, investment bankers, and wanna-be ghetto playas cruising around in dropped escalades with chrome spinnas. It's really quite a shocking scene to behold. It's kind of strange that both the rich trash and the poor trash seem to be attracted to Kirkland on a warm summer eve. It's like they can all smell the common pheromones of empty skulls and rotten souls and are drawn together by it.

07-17-09 | Bicycle Randoms

Single Speeds are currently the SUV's of the bike world. That is, they're ridiculously cheap to make, and yet many of them are selling for *more* than geared bikes.

Bicycle components are absurdly marked up; you get a huge discount when buying a full bike; the problem is the wholesale price for a full groupo is very low, but there's some kind of collusion agreement in the business where nobody will sell you the bits for a reasonable price. You can almost do better by just buying an entire bike at a closeout sale, throwing away the frame and keeping the components.

Bike shop employees are such huge fucking dicks. They treat everyone like a moron and are just generally unhelpful and don't address what you came in asking for, but instead try to steer you to what they have in stock. You fuckers, if you would just be nice to me I would come to you and spend lots of money on components. The other day I was trying to talk to a mechanic about building up my new frame and he said "does it have a fork? do you even know what a fork is?" ; umm, okay, goodbye. You fucking cocks. I guess I'll just buy everything online even though I'd much rather go to a local shop, just because I refuse to give you pricks any of my business.

When people ask me about buying a bike I usually tell them to just go to a good local shop and try some bikes, but I'm rather down on that myself. For one thing they just show me the shitty bikes they happen to carry which is not a very wide selection. For another, I don't think you can actually get any kind of feel for a bike from a short test ride like that. Personally the main thing I feel in a test ride is - this stem isn't the length I like, I would change it, these handlebars aren't the width I'd like, I would change them, these aren't the pedals or seat I like, I'd change them, the seat and bars aren't the height I like, I'd change them. All I can tell is that it's uncomfortable because I haven't adjusted it just the way I want, it doesn't tell you anything about the bike fundamentally. Of course any brand new bike should shift and brake perfectly, if it doesn't it just means the mechanic put it together wrong.

I just bought an old Bridgestone 550. It's a nice steel frame, triple butted Tange CroMoly, but it's not a collector's brand so it's very cheap unlike old Colnagos or something which are just ridiculous. I looked into just buying a new steel frame, but I couldn't find what I wanted. I'll review a few options :

Torelli Corsa Strada is very nice and only about $800. It's not lugged so it's not super sexy, but the big problem for me is it has no down-tube braze-ons. I guess nobody uses downtube shifters any more except that's exactly what I want right now. About $800.

Tommasini Sintesi ; very nice, a bit overly blingy for my taste; more importantly - just absurdly expensive. Almost $2000 for just the frame. For that price you can get a very nice hand-made full custom bike from a framebuilder here in the US. If you buy this you are dumb. (I could say the same for new Colnagos or De Rosa). (there are tons of great US framebuilders, Richard Sachs is just one doing nice work).

SOMA Speedster / Stanyan looks nice, it's pretty, lugged, cheap ($800) - but then you start to look at the geometry - WTF did they do ? It's a weird shape, it's super low, the head tube is tiny. You have to run a very high elevated stem and seatpost; it seems to be designed that way intentionally, WTF. The Smoothie seems like the only decent SOMA bike but they are very ugly IMO.

The Gios Compact Pro is a gorgeous gorgeous lovely Italian frame, not too crazy at $1250, but again like the Speedster once you look into you go "WTF" ? It has a weird "Criterium" geometry with a steeper seat tube and a taller shorter frame. No good. Bikes should be 73/73 only!

Mercian in the UK makes some really lovely new classic-style steel frames. They do everything right, they have the good classic geometry, all the right braze-ons. I think their prices might be reasonable if you're in the UK, but with the conversion to the USD they're too much.

List of things I have to do to my bike :

Remove BB, Fork & Headset
Degrease frame
Get rear dropouts spread to 130
Overhaul headset, new grease and bearings, replace if too bad
See if new cartridge BB will fit
Brush off any rust on frame
Rust proof frame
Touch up frame paint with clear coat
Wax frame
Put on components

07-17-09 | WSOP Spoilers

(The final table is set but won't play until November)

Phil Ivey makes the final table of the Main Event; maybe the best WSOP by any player ever considering how much bigger and tougher the fields are now.

Lots of the pros are twittering their WSOP this year, but Ivey's too fucking cool for that. He hired some underling to twitter for him.

Ivey has some 100-1 bets on himself winning the ME ; like his last props these just seem ridiculous -EV for him, but once again here he is. Rumored to win around $15 M in props if he wins the ME from Andy Bloch, Howard Lederer, Durrr and others.

In other news, chipleader redneck Darvin Moon who got in on a satellite is refusing to wear logos or take endorsements, purportedly giving up about a million dollars.

The cash games in Bobby's Room have supposedly been going nuts. Gus is believed to be the big PLO fish and huge $2k/$4k PLO games are running around him.

Durrrr has issued a live challenge

07-16-09 | Thursday Randoms

It occurs to me that eBay has no way of preventing you from colluding on auctions, having a friend come in and bid with your money to kick up your price. Yeah if you did it over and over many many times with the same people they would pick up on it, but as long as you change accounts or just don't do it a ton they can't stop it.

When I toured the house I'm looking to rent, it had the previous tenant's posessions; it was full of the dishevlement of family life, child's toys. It was lovely. I want that so bad. A single person's home is never pleasantly cluttered, it's either spare or just messy.

I think Facebook is a disgusting waste of time, it's social masturbation, electronic gossip, and since nobody interesting has ever written anything of any use on Facebook, I have no reason to read it. However, I am very glad that Facebook exists, because it voluntarily sequesters the retards into their own corner of the internet that I never have to see. It's kind of amazing and wonderful how people do that to themselves; for example in life you see people with tribal arm bands, or pleated pants, or huge metal plugs through their nostrils - of course they're awful things to do to yourself and look ridiculous, but it's wonderful that the retards choose to do those things, because it instantly and overtly labels them so that you can easily just avoid them.

God damn I hate synthetic fabric. I went to buy a new comforter at JC Penney since I've had the same one since high school. 90% of them are some awful shiny crinkly weird feeling synthetic bullshit. I want to only touch cotton and wool and wood and flesh. I want to only smell dirt and coffee and sweat and grass.

07-15-09 | Wednesday Rambles

Printers should just be on WiFi. In fact pretty much everything except video and power should be (speakers for example).

I got my standing desk set up. I'll post some pictures / progress report at some point. It really annoys me that we don't have a standardize docking multi-port. When I take my laptop from home to my sitting workstation at the office to my standing workstation, it should just be one cable that I plug for video, power, USB, etc.

Fucking god damn you web programmers suck so bad. When you put "Seattle WA" in the box for Expedia, you get a STOP! error screen saying : "We could not find any airports that match your search for seattle wa. We found the following airport with a similar name. Leaving from: Seattle, WA (SEA-Seattle - Tacoma Intl.)" . Umm, okay, seeing as there are NO OTHER CHOICES that are even close, maybe that is in fact the one that I intended, hmm ??

Obviously this case is just broken, but as a general point of computer program UI interaction design, in cases of semi-ambiguity like that, the right thing to do is not to have a modal "STOP" screen, but just rather go ahead and make your best guess and proceed, but provide a simple back up button. So when the user puts in something that you're 90% sure you know the answer to, just go ahead and immediately show them answer, and put a little button in the corner for back up & refine.

There are two points of this UI design that I think are crucial : 1. if you provide a fast path that is accessible in a predictable way, users will modify their own behavior over time to adapt to the fast path; they will learn patterns of use that make them more efficient, you should design the UI such that it is amenable to that, and 2. you should *never ever never never ever* have different sequences of screens. That is, the pattern of prompts and displays must be 100% deterministic - it should not be conditional. Over time users learn the pattern and will start typing without even looking at the screen. They can go much faster if it's a predictable rhythm, of prompt A - B - C. If you sometimes pop up another prompt - A - D - B - C depending on what they entered in A, they will be doing the B typing in D and it's all fucked up. Don't do it.

(I get similar stuff for Google Maps all the time. Like if you type in "la guardia" it gives you a "did you mean "la guardia airport" or "laguardia airport" ? and the two of them have location tags in the exact same spot. Actually the more I use Google Maps the more it pisses me off. The thing that sucks about it is that it's very modal, but it doesn't expose the modes to you. It has these states like "searching for address by name" or "searching for businesses near address" but once you get in one mode you can't switch from one to the other in any easy way. I wind up having to copy-paste the addresses and then go back to home and back to maps again to clear the state.)

I signed on the stupid house; I just know I'm gonna get fucked and in 6 months I'll be whining on here about how I have to move again, but I'm just so fucking sick of looking at places right now. It's dumb and irrational but I just want to get all these distractions out of my life so I can clear my head and focus on work. It's pretty much the way I make all my major decisions in life - I'm super picky and indecisive until I just get sick of looking and then make a snap irrational decision to take the first thing I see.

There's a house near my current apartment under construction, and also one near the new place. Both of them are these fucking amateur asshole productions where they are just taking forever. The one by my current house they literally work on maybe one day each month; it's literally been under construction the entire year that we've lived here and there's zero visible progress, they've worked on it maybe 20 days out of the whole year. And they always come from like 7 AM - 10 AM and then leave. And it's always just one guy, so he never gets anything done other than making a bunch of hammering and power saw noises to wake me up in the morning. WTF there should be laws against this shit. Hire a fucking proper crew and you could get the whole thing done in a week and be done with it and stop torturing your neighbors. I hate the nanny state explosion of stupid laws about everything, but every time you don't have a ridiculous law, people are fucking assholes. You have to dictate explicitly every single rule of basic behavior. (I know part of the delays with construction is probably the permit process which is in fact caused by the stupid permit laws - however I'm sure if we didn't have all the construction certification permit laws then people would build structural walls out of cardboard because they are fucking greedy retards who will fuck you in every way possible unless you force them not to).

People get the wrong idea of my mood from my blog. I'm actually very happy up here since summer hit. I'm spending tons of time outdoors in the sun, which = instant happiness for me. When I'm happy, I'm out exercising, running around. The only time I come in and sit at the cursed computer and write this nonsense is when I'm not happy. You pretty much get my unloading of vitriol. It's a sampling bias problem - the thoughts I choose to write here are not a random sample of my thoughts.

07-11-09 | Saturday Rambles

I bought some underwear on Amazon, and now my main page "you would like this product" is full of closeups of guy's crotches. Yay.

Man it would be sweet to get those rub-down type massages that pro athletes get (mainly boxers and cyclists that I'm aware of, where they really grease you up and rub out the lactic acid). I've never known a "massage therapist" that did that.

When I was biking, I needed to adjust my seat, so I pulled over; I didn't have all the tools I needed, so as I was fiddling around, some guy rode by and yelled out "need help?" and I yelled "yes!" ; he came back and gave me an allen wrench and I got it done. The wonderful thing about cycling is that this is not really remarkable at all - cyclists always are just *human* to each other - you try to be considerate of other cyclists, don't cut them off, you ask if they need help when someone is stopped, etc. It's so pleasant and encouraging to me, as opposed to every other human interaction I have where people are just douches and dicks and manipulating and lieing and at each other's throats. I'd love to cut out all the things I do where people are rotten to each other and only do the things where people are nice to each other. So that means no dating, no pickup, no buying, no selling, no contracts or anything where you might need customer service, no driving. I'll just bike, go hiking (the day hike weekend warriors are total dicks who cut across switchbacks and don't yield to ascenders, but the serious hikers who you meet way deep in the woods are always super nice), and ... that's it, I can't think of any other activity where people are nice to each other.

Yay! The lake here is finally warm enough to swim in. The life guard lady yelled at me. Apparently you have to stay within 50 feet of shore. WTF. I've seen guys swimming right across the lake before. You can easily enough find lake shore with no lifeguard; the little fake sandy beaches get crowded, but there's lots of grass frontage on Lake Wash and you can jump in anywhere there (getting back out is a bit trickier). One problem is there is some very thick seaweedy type stuff (I guess it's not seaweed cuz it's fresh water, but underwater plants) that comes up almost to the surface. In the official swimming areas they cut that stuff out, but in the guerilla swim you can get tangled up in it and it's very hard to move; it feels like tentacles actually grabbing your arms and legs trying to hold you back.

I'm thinking about building up a bike on a classic steel frame with modern components. It would be really expensive, because individual components are absurdly overpriced, you get a big discount buying bikes as whole pieces, and it would take a lot of time (it would be so much quicker and easier if I had a full shop with a stand and cable on spools and whatnot). But it would be a fun project, and then I would have a bike that I was intimately connected to when I'm done.

It sucks being bigger than average in some ways (most of the time it's good). One is at thrift stores, anything good I find is always too small for me. Another is buying used bikes. There are shit tons of amazing great bikes on ebay in the 54-57 cm range. For me (58-61) there's very little.

Colnago is one of the premier steel frame manufacturers, but they make the ugliest paint jobs known to man; it's like they got some high school student from 1984 to air brush their frames; I totally expect to see a unicorn in space on their frames; ( example or example ). The people who think Colnago frames are beautiful are the same guys who wear those weird baggy pants to lift weights. ( Apparently they're called Zubaz Pants ; WTF ). A lot of bike bar tape and Zubaz pants comes in this random color speckle pattern that I think of as someone eating a bunch of skittles and then throwing up. ( oh, here's another Colnago ; it looks like a unicorn took a rainbow shit all over it )

( BTW you can also substitute rainbow barf or rainbow piss )

this is bicycle perfection; Italian steel with a flat top bar and chromed lugs. (the paint job on the frame could stand to be even a little simpler; I hate all the big badges). They are absurdly expensive though, around $2000 for just the frame. Even if that weren't a lot of money to me, I couldn't pay that just out of moral outrage. ( the Classic Bike Shop has a load of gorgeous frames ).

Urgg I really shouldn't be buying bikes right now since it's the peak of summer and the worst possible time to buy, but I'm just loving riding so much and I got a bee in my bonnet that it would be fun to build up a new bike and now I want to play. If you're smart you buy in December.

I think the US is the only 1st world country that still uses dry butt wiping technology. Come on, we're the freaking Roman Empire of the modern era, we should have bathroom slaves to lick our butts clean ! Or at least washlets. Hmm, I've been watching the HBO Rome too much. It's pretty great, so far as trashy soft-core porn soap operas go. It's great for vicarious living fantasies; I'd love to have hot naked chicks scattered around my house all the time, and when upstairs neighbor starts clog dancing at 1 AM I'd just grab my sword and go up and stab him in the neck.

There're a lot of gay guys around Cap Hill here that are just absurdly skinny, very fit, bugling muscles, but with taught skin and bones poking through like a starving African. I hope they just have AIDS and aren't doing that to themselves intentionally, cuz that would be really fucked up.

Honesty in selling is never rewarded. People might consciously think "I appreciate he's being up front about the drawbacks" but they will still go with the liar. This comes up with us at RAD selling middleware. We're very honest about our products and tend to be realistic and limited in our features; many of our competitors promise the kitchen sink but deliver a non-functioning mess. Still, a buyer looking at a feature list that says "just X" or "X + Y + Z + W" will often choose the one they know is too good to be true. Fortunately for us we have Mitch and a great reputation, but you see it all the time in corporate purchasing evaluations. Of course it also comes up in dating and selling yourself. If you're honest about your drawbacks right from the beginning, that turns them off and it blows it. If instead you keep quiet and get them hooked and then much later reveal your problems, you win, it's too late for them to back out. Like if on your first date you just say "you know, I'd like to fuck you, but you're just too dumb for us to have a serious relationship" it's over. The liar is always rewarded. Similarly, people never tell you about STD's. You might think you're being careful and trying to wait until you know someone so you can trust them. HA! They might tell you *after* you have sex that they have HPV or Herpes but just don't have an outbreak right now. Gee thanks. They know subconsciously that if they say anything before you will not give them credit for their honesty, rather you will just flee.

I was thinking about this because of the house I'm renting. The owner has been up front with me that they might have to move back in to the house next year. They didn't need to tell me that, by law they can just do it whenever they want. If they were a real dick they wouldn't have said anything and I would've just moved in. Instead they were up front, and while in an abstract rational sense I respect them for that, in a real sense it makes me not want to move in, and basically just winds up punishing them for their honesty.

BTW the house rental situation also makes me think of something I've written before - the right way to handle promises of contingencies is by betting. Say the owner tells me there's a 10% chance that they will move back in. There's a very easy way to force them to be honest about that percent - just make a bet. I put up $1000 , they put up $10,000. If they were honest about the percent, it's neutral EV, nobody profits on average. But if they were lying and the real chance is more like 25% or 50%, then I make a profit on average. Of course nobody is rational enough to agree to something like this.

07-10-09 | Friday Rambles

ZOMG the traffic here has reached whole new levels of clusterfuck since the I-90 bridge is closed for repairs. I worked at home the last two days to avoid it, but decided to come in today for a change of scenery and am seriously regretting it. (ADDENDUM : ZOMG I just looked again, I'm gonna be here until midnight, sigh sigh sigh).

One of my problems with working at the office is that I always seem to forget something that I want/need at home or vice-versa. When I went home I accidentally left some papers I wanted to read here. Today I forgot to bring in my swim suit so I could swim over lunch. URG. The bigger problem is I just feel like I can't relax at all ever the entire day at work. When I'm home I'll work intensely for a few hours, then just take a break and really relax, have some food, watch TV, take a walk, then get back into it. I can take breaks at work, but they're not relaxing breaks, I never unclench. I get a burst of productivity in the morning, but then I'm drained and I never refuel and the afternoon is just a waste of staring at the code and poking at the keyboard and nothing really happening.

The Nissan salesman isn't returning my calls & emails. WTF WTF. I want to boycott all you fuckers who try so hard not to sell me things, but then I would just boycott everything.

Sea Breeze Farm sells at Seattle Farmer's markets and seems to have some amazing pork products. More investigation is needed.

Burning Beast is this weekend at the Smoke Farm. here's a video . Meh it looks a bit lame but I do like the idea of a big food festival on an old farm.

So I might have found the house to rent here . It's not in exactly the spot I want, but it's not bad (0.6 mi to the action). I'm putting down an application anyway. It's pretty great (the kitchen is fantastic - big butcher block counters, gas stove, lots of light) and it has a porch; I want to drink iced tea and say "I say I say" on my porch. Some of the neighboring houses look a little sketchy, like they might have students in them, which is worrisome, but it's hard to ever know what your neighbors will be like. (I'm vaguely considering going and knocking on the door to talk to the current resident and ask them what the neighbors are like).

Anyway, this rental experience has got me mildly outraged. First of all the realtor was half an hour late to our appointment to see the place. I'm no longer shocked at sales people's ridiculous absurd lack of professionalism and courtesy to clients. Okay, let's move on.

The thing that's really absurd is the owners might need to move back in. Apparently they work for MSFT overseas and might get summoned back here. They find out on August 15, but they are trying to get someone to take a lease that starts August 1. WTF WTF that's just absurdly greedy and dickish. How could you not just wait to lease it until August 15 !? In order to avoid losing 2 weeks, maybe a month of rent, you want someone to move in and then perhaps immediately move out again. That's just so selfish and unreasonable. So I'm putting an application in, but I'll have to talk to them about this moving back issue and be a total dick about it and get something in writing. (though I'm not sure if anything even in writing protects me - I think in WA homeowners have unconditional right of return to evict tenants and move back in)

In other news : Amazon wins. That's it, any doubt about who would win the internet shopping wars, or whether Amazon adds value as an entry point is over for me. Every other shopping web site I've used is just so absurdly broken, I refuse to use anything else. Cambria Bike has been really slow shipping me stuff and their site is so slow and their web forms and account management are so fucking janky, it's infuriating. It's so much better to just buy bike stuff from merchants that sell through Amazon. Even people who have their whole own online store I'd rather go through their Amazon store (LL Bean for example - I like their pajamas cuz you can get tall sizes, and they're actually pure cotton instead of fucking douche-atrine neo-polymer-crinkle-stretch-shine-azin). Which brings me to another rant :

I HATE CAPTCHAS !! OMG I hate that that fucking dumb douchebag got a god damn Genius grant for the dumbest fucking most obvious god damn idea in the fucking universe. I seem to spend half my damn time typing in catchas now because I have a lot of trouble reading them myself, and I'm never sure if the damn page needs me to capitalize correctly or not. I hate "recaptchas" even more because the scanned words are often super illegible. The worst thing is that I would say 90% of pages that use captchas are horrifically fucking broken. I type in some fucking annoying web form, and then there's a damn captcha to submit, and of course I fuck something up in the captcha, AND IT CLEARS THE DAMN WEB FORM !! ZOMG ! Oh it's so angering. Fuck you captcha guy, you should get the MacArthur Knee to the Groin Award.

On that note - YARG the fucking "confirm your email address" box is so tilting. Hello, I fucking copy-paste to fill it in (oh, which reminds me, I have found a few sites that make the confirm box with a flash widgets and forbid copy-pasting in it; that's really fucking dickish). The worst thing is just like captchas, so many damn sites will just wipe out your whole form if you hit "submit" and you left the "confirm your email address" box blank.

I finally got out biking yesterday; it had been over a week due to my mom being here and then a spell of shitty weather. I had a little spell of depression again, and desperately needed my visit to the 30 mile psychiatrist. There's this wonderful metaphorical doctor in my body who literally pumps liquid happiness into my veins around mile 30. If I don't see him regularly it's like going cold turkey off Prozac and it's not a good scene. I've never found any other activity that so reliably gives me relief from my congenital gloom.

On the minus side, the fucking awful pot holed roads here are literally destroying my beloved expensive bike. My head set is grinding a bit now, presumably because the bearings are getting damaged, and I've got a ding in front rim. Fuck. I was thinking of just driving to Mercer Island to just ride there, but of course you can't do that now with the 90 closed. Blurg. Also, a lot of the worst pot holes are of course on the right edge of the road where the bike is supposed to be, so I get the unpleasant choice of swerving out into traffic to avoid the pot hole or slamming into it. (sadly cars generally only give you enough room for the size of your bike, which completely traps you into a straight line, and is really not enough room; you need a few feet to be able to swerve to avoid road obstacles). That's literally the scenario that made me crash and separate my shoulder back in SF (wiping out on a pot hole because I couldn't avoid it due to cars all around).

Seattle in the summer is glorious.

07-07-09 | Small Image Compression Notes , Part 2

Deblocking survey :

There are a few different ways to come at the problem theoretically.

One is to work on post-decode data in spatial domain. These approaches basically work by explicitly trying to detect block edges and just filter them. This is the approach, for example, of the H264 "in loop deblocking filter" which there is a lot of literature on. See for example "Adaptive Deblocking Filter" by List, Joch, et.al. For an example of the filter-based approach on the 8x8 DCT case see "DCT-Based Image Compression using Wavelet-Based Algorithm with Efficient Deblocking Filter" by Yan and Chen. (BTW the JPEG standard contains a "block smoother" which basically predicts AC1 as a linear function from neighboring block coefficients. This is okay for the specific case of smooth images and very high quantization, but is generally not awesome and is an ancient technique. Ignore.)

A more hardcore version of the filtering approach is "Combined Frequency and Spatial Domain Algorithm for the Removal of Blocking Artifacts" which does adaptively-offset and adaptively-directed gaussian filters ; this is sort of like the image denoising stuff that creates pixel gradient flow vectors - the filters are local gradient adaptive so they don't go across real edges. This appears to perform quite well but is very expensive.

The other general approach is a more abstract maximum-likelihood idea. You received a lossy compressed image I. You know the original image was one of the many which when compressed produces I. You want to output the one that was most likely the true original image. This is a maximum likelihood problem, and requires some a-priori model of what you think "natural" images look like. In particular, for the case of quantized DCT coefficients, you have a quantized DCT coefficient C ; instead of just reproducing Q*C you can reproduce anything in the range { Q*C - Q/2 , Q*C + Q/2 } , and you should choose the thing in that range that makes the "best" image.

"Optimal JPEG Decoding" (1998) by Jung, Antonini, Barlaud takes this approach directly. Their results are not awesome though; presumably because their prior is not good. A more modern version of the same idea is "Block Artifact Reduction Using a Transform-Domain Markov Random Field Model" by Li and Delp which uses a better model for image likelihood, but is in the same vein of doing a brute force search in the allowed coefficient space to find the maximum-likelihood reproduction.

A related method that was popular for a while is "Projection onto Convex Sets". This is basically just a method of satisfying simple convex constraints in an optimization. Here our constraint is that the quantized coefficient stay the same, that is, repro in { Q*C - Q/2 , Q*C + Q/2 } . You then apply some target function, such as you want smoothness or something, and take iterative steps towards that goal and project onto the constraints one by one. There are a lot more details to this, I haven't paid too much attention to it because these are all crazy expensive and I want something realtime.

"Blocking Artifact Detection and Reduction in Compressed Data" by Triantafyllidis etal (2002) is in the same vein but simpler and more analytical. It again worse directly in DCT space on coefficients within their quantization range, but it directly solves for the ideal reconstruction value as a function of neighbors based on minimization of specific simple deblocking metric. You wind up with just some equations for how to modify each coefficient in terms of neighbor coefficients. While the paper is good, I think one of their base assumptions - that the frequencies can be dealt with independently - is not sound, and most other people do not make that assumption.

"Derivation of Prediction Equations for Blocking Effect Reduction" by Gopal Lakhani and Norman Zhong (1999) is an older, simpler still version of the Triantafyllidis paper. They only correct the first few coefficients and solve for optimal reconstruction to minimize MSDS (mean squared difference of slopes). You can actually look at the equations here and they're very intuitively obviously right. For example, the first AC coefficient should be corrected using the difference of the neighboring DC coefficients. In case you don't see that that's obviously right, if you have DC's like [8],[16],[24] after dequantization at Q=8, and your AC's all got quantized to zero, obviously the original image most likely had a smooth slope, so the first AC in the middle block should be predicted to be the linear interpolation.

An interesting one I found that's related to the stuff I tried with smooth reconstruction of the DC band is : "Improvement of DCT-based Compression Algorithms Using Poisson’s Equation" by Yamatani and Saito (2006) .

BTW a related issue that often comes up is the incorrectness of center dequantization of AC coefficients. I've written about this before and lots of these papers mention it; the best full note on it is : "Biased Reconstruction for JPEG Decoding" by Price.

The very modern stuff has gotten quite arcane. People now are doing things like directional overcomplete wavelets on the reproduced image; with this they can detect both block artifacts and also ringing and other quantized transform artifacts. They then use maximum-likelihood markov models to guess what the source image was that produced this output. This stuff is extremely complex and I haven't really followed it because it's nowhere near realtime, but probably the best solution for offline very high quality JPEG decoders.

An interesting outlier is John Costella's Unblock . It's based on a clever simple idea that I've never seen anywhere else. Unblock is based on the assumption that pixels near the block boundaries come from the same model as pixels in the centers of blocks. That sounds obvious but it's quite profound. It means that pixels near the edges of blocks should have the same statistics as pixels in the centers (in the maximum likelihood lingo, this is a prior we can use to choose an optimal output). In particular, it's useful because in the DCT the interior pixels are much more accurate than the edge pixels. What Unblock does is looks at the statistics of the decompressed interior pixels and assumes those are our goal, and then it forces the pixels near the edge to match the statistics of the interior. The corrections are applied as wide smooth filters.

07-07-09 | The World Is Horrible

I took the Montlake ramp this morning, which I never do, cuz I had to get gas. It's one of the ramps that gets all jammed up with traffic and there's a metered light. As I'm sitting there slowly losing what small supply of patience I had available for the day, I watch single driver cars go by in the carpool lane. First a mercedes went buy, then a BMW, then an Audi RS4. I was think "rich douchebags think they're entitled" but then a Chevy and a Pontiac went by and my stereotype was ruined.

Some days I feel like I've done all the "work" I can stand in the first 30 minutes of commuting (write tech/algorithm code is not really "work" , though dealing with MSDev or cdep project settings is "work", as is fighting the damn broken IEEE web site to give me the papers I paid for). Yes, yes, I know I'm a big whiner, there are poor people who take ten buses to commute for three hours to their shitty job at McDonalds where some pimply teenage assistant manager treats them like garbage all day long, the customers are rude to them, and they breathe oil fumes under flourescent lights.

Anyway, I pull out of the ramp and step on the gas to rapidly get up to speed and merge smoothly, as I always do. After a few seconds I notice there's a cop right behind me. He follows me closely over the bridge and then puts his lights on to pull me over. Hmm.. I'm not really exactly sure what he's going to say. He comes up to the car and says "I saw you accelerate out of that ramp and rapidly change lanes". My jaw just drops. Uh, yeah, that's what you're fucking supposed to do dumbass. I say "okay". Wow. I'm in shock, I feel like when the guy asked me not to park in front of his house on the completely empty street, my mind is blown. He winds up writing me a ticket for going 65 (I have no idea how fast I was going, there was lots of traffic, I was just going the same speed as the guy in front of me). After he writes the ticket he tells me to be safe pulling back into traffic, so I say "so you want me to get on slowly and jam everyone up?" ; he gives me a dirty look and says "just be safe". (I try hard not to sass cops anymore; I got my car towed once because I told off a cop that pulled me over for expired registration; it's safer to sass them after they've written the ticket because then they'd have to do a whole new set of paperwork to write a new one). (I was particularly aggravated because I'd just been watching all the fuckers cheat the carpool lane).

Ugh. I certainly don't deny that I deserve speeding tickets, I speed my balls off. The funny thing is I never seem to get them when I'm actually speeding. I got one for going 60 mph on the freeway in Houston. I got a ticket in Seattle ten years ago for "unnecessary tire noise" when I went around a corner - this in a fucking Prelude (I told the cop at the time, dude I couldn't spin my tires if I wanted to). The only time I actually got a ticket when I was speeding was out in the country in Texas near Sugarland in a bunch of empty roads for a subdivision that hadn't been built yet. (it was off the 99; I guess it was probably "Long Meadow Farms" or "Waterside Estates" or something, though none of that existed at the time). Houston suburb construction is incredibly corrupt; the big builders all have connections in government, and the roads and power and sewer lines are laid in advance and paid for by the city, effectively a huge subsididy for these suburb builders who keep pushing ever farther out into undeveloped land. I'd found a patch of roads that had been laid out into fields that had only months ago been that dense nasty Texas woods, and was speeding my balls off around the empty roads; I got a ticket for doing 85 in a 45 there which normally would be a license-suspending kind of thing but I guess the cop had a little bit of mercy based on the fact that there was not another human in sight for miles around.

In my youth this ticket would have made me outraged, angry at society, and talking of all kinds of drastic reactions. Now it just makes me sad. I'm tired.

I watched the movie "Stroszek" a little while ago. It's pretty terrible, but it had some great moments. It's a Werner Herzog movie, so it's a total mess. Herzog movies are sort of like the film equivalent of experimental free jazz. He just sort of has a vague idea and tosses some people in front of a camera and sees what happens. Yes, they are as terrible as that sounds, but there are moments of brilliance, and even when they are failures at least they're interesting and full of humanity. The best scenes in Stroszek IMO are when the banker comes to foreclose on the house and Bruno rants the whole time about how this bastard is crushing his life while doing it with a smile. All Herzog movies are about the struggle of individuals to claw a way through this world with a bit of self-determination, dignity, and individuality.

It looks like my cable is fixed; I'm still not sure exactly what the problem was. The technician who came out replaced a bunch of lines, but I had had zero problems for a year and then suddenly had problems all the time so I don't really believe that bad lines were the problem. I assume that something changed upstream which made the signal to my building worse, which then pushed me over a threshold where my bad lines started causing problems. I've heard this can happen when the cable company adds a bunch of subscribes and splits an upstream wire, it lowers signal quality for everyone.

Anyway, I wanted to write some notes cuz the technician said some things I didn't know. One was to not use a signal booster. Since my SNR was shitty before I was running a Motorola high-bandwidth signal booster, which fixed my problem with squiggly lines and noise on my TV. Apparently the cable modem doesn't like it, because the modem only works in a certain power band, and I was actually giving it too much power. The other thing was that one of my interior lines was an old wire. Apparently they've changed the wires, the newer wires can carry a wider signal spectrum. I guess that's part of how they keep upping the bandwidth they can send on the wires - higher frequencies.

07-06-09 | Small Image Compression Notes

Lapping appears to be a complete red herring. I've wasted a lot of time on it and I'm very angry. I've been trying to work up a lapped block DCT image coder. The idea is that block-DCT-based is good for speed and parallelization for micro-core architectures, good for memory bandwidth, etc. and the lapping theoretically lets you avoid some of the nasty block artifacts by effectively extending your basis functions.

In practice it just doesn't work. I've tried lots of different lapping methods, and in all of them if I make a parameterized lap amount based on a kaiser-bessel-derived window and then tweak the lap amount to maximize SSIM, it tunes to no lapping at all. Basically what's happening is that the extra bit rate cost caused by the forward lap scrambling things up is too great for the win of smoother basis functions on decompress to make up. Obviously in a few contrived cases it does help, such as on very smooth images at very high compression. (of course the large lap basis functions are a form of modeling - they will help any time the image is smooth over the larger area, and hurt when it is not).

The really retarded thing about this is that areas where the image is very smooth over a large area are the cases we already handle very well!! Yeah sure naive JPEG looks awful, but even a deblocking filter after decompress can fix that case very easily. In areas that aren't smooth, lapping actually makes artifacts like ringing worse.

The other issue is I'm having a little trouble with lagrange bitstream optimization. Basically my DCT block coder does a form of "trellis quantization" (which I wrote about before) where it can selectively zero coefficients if it decides it gets an R/D win by doing so. Obviously this gives you a nice RMSE win at a given rate (by design it does so - any time it finds a coefficient to zero, it steps up the R/D slope). But what does this actually do?

Think about trying to make the best bit stream for a given rate. Say two bits per pixel. If we don't do any lagrange optimization at all, we might pick some quantizer, say Q = 16. Now we turn on lagrange optimization, it finds some coefficients to zero, that reduces the bit rate, so to get back to the target bit rate, we can use a lower quantizer. It searches for the right lagrange lambda by iterating a few times and we wind up with something like Q = 12 , and some values zeroed, and a better RMSE. What's happened is we got to use a lower quantizer, so we made more, larger, nonzero coefficients, and then we selectively zeroed a few that took the most R/D.

But what does this actually do to the image qualitatively? What it does is increase the quality everywhere (Q =16 goes to Q=12) , but then it stomps on the quality in a few isolated spots (trellis quantization zeros some coefficients). If you compare the two images, the lagrange optimized one looks better everywhere, but then is very smooth and blurred out in a few spots. Normally this is not a big deal and it's just a win, but sometimes I've found it actually looks really awful.

Even if you optimize for some perceptual metric like SSIM it doesn't detect how bad this is, because SSIM is still a local measurement and this is a nonlocal artifact. Your eyes very quickly pick out that part of the image has been blurred way more than the rest of it. (in other cases it does the same thing, but it's actually good; it sort of acts like a bilateral filter actually, it will give bits to the high contrast edges and kill coefficients in the texture part, so for like images of skin it does a nice job of keeping the edges sharp and just smoothing out the interior, as opposed to non-lagrange-optimized JPEG which allocates bits equally and will preserve the skin pore detail and make the edges all ringy and chopped up).

I guess the fix to this is some hacky/heuristic way to just force the lagrange optimization not to be too aggressive.

I guess this is also an example of a computer problem that I've observed many times in various forms : when you let a very aggressive optimizer run wild seeking some path to maximize some metric, it will do so, and if your metric does not perfectly measure exactly the thing that you actually want to optimize, you can get some very strange/bad results.

07-06-09 | Readiness

Went to Ballard Sunday and had a few amusing parking related incidents.

One was the famous street parking nittiness that I've heard stories of but never seen in real life. I was trying to take my mom to the Nordic Heritage Museum. I guess we turned on the wrong side of it so we were on the wrong side of the block from the parking lot, so I figure I'll just park on the street, no biggie. I park and we're walking towards the museum slowly, when a man bolts out of the house I parked in front of and covers over to me; he says in that standard meek/confrontational way "excuse me, sorry, can you move your car? it's just you parked right in front of my house.." ; I was like in complete shock, my mind is blown. You have to realize the streets here are like completely empty, there are like maybe 10 cars on the entire block with 100 spaces on it; hell the curb in front of his house has room for two cars and there are no other ones around. I'm just like "really?" he starts going into "blah blah blah" so I'm like "fine fine it's no big deal yeah I'll move it".

I've never encountered the amazing "not in front of my house" nittiness. It blows my mind that this guy is sitting around all the time with his ass tied up in knots keeping an eye out for anyone taking his precious curb.

BTW "nit" is a poker term that I think should make it into general parlance. It obviously derives from "nitpick" but the meaning is more specific. A nit is someone who's obsessed with technical correctness, the absolute by the book "right" answer to a question, to the point of being very annoying and missing the big picture. Often people who play too tight are referred to as nits, but it's more general than that. For example if you ask someone "how do I convert a stdio FILE to a file descriptor I can use with fcntl in the C standard library?" and they say "well, I dunno, cuz there is no fcntl in the C standard library, that's in the POSIX library" , that person would be a nit, and they said would be nitty. Lots of people on comp.lang.c.moderated are super nits; lots of grammar nerds are big nits.

The other parking incident was at the locks. It was a pretty day out so the parking lot was all jammed up with tons of people. I fucking despise waiting in parking lots, and there's a ton of easy street parking around there, so I wanted to just bail out and get out of the lot and go park on the street, but there was some dude in front of me waiting for a car to back out so he could have the spot. I swerve around him to get past so I can exit, and just as I do I see another car is also pulling out (in addition to the one he was waiting on). I stop and ask the first guy if he's leaving, he says yes he is, so I know there are two spots leaving, so when one car pulls out I take the spot. Then the second car pulls out, and the original waiting guy takes that spot. I'm thinking okay, everything is hunky dory, but as I walk past the guy who was waiting (who is now parked) I hear him say "asshole" in that lame quiet way so it's just barely where I can hear it. It just startled me. I mean, for one thing it blows my mind, how could you not have noticed that I stopped at specifically made sure there were two spaces before I took one? How can you possibly be all upset about this incident, and then to drop the lame quiet asshole bomb, I just wasn't ready for it.

I have a lot of trouble with these surprise confrontations. I need a lot of time to get ready for human interaction; I have like a 5 minute delay loop or something before I can process a situation. There are times when I'm all pre-angered and ready to burst at anyone who gives me trouble, but I have to prepare for that, I can't just do it on the drop of a hat. 5 minutes later I was ready to tell the guy off, I was wishing I would see him again around the locks and he would say something so I could tear into him, but of course it didn't happen. This same delay has always given me a lot of trouble with girls too. If I really actively get myself into socializing/flirting mindset, I can do okay, but it takes a good effort of preparation to clear my mind and give myself pep talks and whatnot. If I'm just going about my daily business, like if I'm grocery shopping or something and girl says "hey those are really cute shorts" it just startles me, I'm not ready for it, my response is like "umm, hmm yeah" or something, and it's not until five minutes later that I realize "oh, she was flirting".

07-04-09 | Faux Pas

There are quite a few people around me who have extremely unpleasant personal comportment ticks. Little sounds they make, or something they do while eating, or some minor habit of the way they do ordinary things. This is an example I made up to specifically not be a real one of someone around me : something like slurping loudly every time they sip a liquid, even just water or anything.

It's not a huge deal, but it's really grating and sort of gross, and it's also frequent so that every time they do it I think "urg, should I say something? no, if I do they'll just think I'm a jerk and won't actually see that it's a really abnormal and bad habit that they have". Nobody ever actually wants you to tell you anything real about themselves; even if they are grateful that they learn it, they will still hate the messenger.

It makes me wonder/worry about what kind of disgusting habits I have that I don't know about because noone tells me.

I've always wanted to invent a backpack that didn't make your back sweaty, but it's rather a difficult issue. You certainly could have a mesh box frame between your back and the pack to let air through; that would also give you a more consistent padded layer. Basically something like a whole bunch of little semi-soft whiffle balls.

The other thing I really want is a better ice pack. Those big block ice packs are really worthless. The ideal one would be just like frozen peas. I think this is quite easy and practical, you just make bunch of small plastic spheres and fill them with that ice pack blue liquid stuff, and put it all in a bag together.

The idea of not going in the sun because you're afraid of skin cancer is very bizarre to me. Being alive is better than not being alive, but not by a big margin, so the idea of giving up something that you really love to slightly extend your expected lifetime just seems like a huge -EV play. The only way living is worth doing is to try to maximize your pleasure, and if you die early, no biggie.

My new doctor basically told me the same thing about my shoulder. He said don't baby it, just go ahead and do whatever you want on it, push it, play sports, lift weights whatever. Yes, eventually it will probably become arthritic and need surgery, but you may as well push it until then. If you baby it, you might delay or avoid surgery, but then you're missing all those years of really using it, so what's the point of babying it? What are you saving it for?

I feel like some people think that life has some huge treat in store for them at some mysterious juncture in the future, so they need to keep their bodies alive and save money and be there for this magic stroke of luck. Well, I got news for you - it doesn't get better, there's no pot of gold at the end of the rainbow, there is no god, no afterlife, no prince charming.

07-03-09 | Hot summer nights

I just discovered that my wifi works from the stoop outside my apartment, which is quite delightful. It's a glorious long hot summer night. Perfect for biking around the city with your new lover, lying in the grass laughing, swinging in the playground, sipping drinks on a patio, getting covered in the smell of grass and sweat and sweet summer flowers.

I adore the heat, but it does make my productivity go down immensely. I just can't work in heat. I'd love to have a house that's super open, like maybe traditional Japanese style where the house is basically just a wooden platform and you have sliding rice paper screens, or even like a deluxe Arabian prince's tent, where you have pillars at the corners and cloth on the sides that you can roll up if you want (and just piles of pillows on the ground to lay in with your harem).

I thought about writing a big rant about cars and bikes and pedestrians, but I got halfway done and deleted it. In the end, it comes down to one question to me : would you rather live in a world where it is a pleasant and relaxing experience to be a pedestrian or a cyclist, or would you rather live in a world where people in cars reach their destination a minute or two sooner ? To me the answer to that question is abundantly clear, and anyone who makes the world less like the way I want it to be is my enemy.

In fact, I don't think it's a matter of "taste" or "preference" or anything. These are copouts that people who are wrong use to justify doing the thing they know is wrong. If you prefer your steak well done, you are just wrong. Now, that's fine with me, if you want to be wrong, you are free to choose the stupid choice. In fact I respect and like people who are self confident and knowingly and unashamedly choose things that are not "proper". However, don't pretend that both ways have equal value and it's just a matter of different "opinions".

I just saw the play "Orange Flower Water" ; it was pretty good, I generally don't like plays, but I do enjoy them under a few conditions - 1. the play must have no "whimsy" or "pizzaz" or men in drag or madcap mixups. 2. it must be a very small intimate theater, not a big production, I want the performances almost in my lap (but not ever actually in the audience), 3. it must be very short. This was 70 minutes which is about perfect. My review of the play : a bit trite and predictable, but the immediacy of the human performances helped make it real; the ending is very cheesy and should be cut; the staging here in Seattle was very good, minimal and clever; two of the actors here were great, one was very poor. (an amusing note : one of them was the voice of Cortana).

Anyway, what I wanted to rant about is "Q&A Sessions". At OFW the playwright was in the audience this night only and did Q&A after. Oh my god I hate Q&A sessions so much. (I would have just left, but it was such a tiny space it would've been very awkward, and I was with my Mom who I think wanted to stay for it). First of all, the main reason I hate Q&A sessions is that I just generally hate knowing the writer's or director's opinions of their own work. I saw the work - the work is its own explanation. There's an awesome quote from David Lynch, maybe someone can find me the exact words, someone asks him to explain a certain scene and he says "the thought that was I trying to express in that moment is what you see, the movie is my way of communicating the idea, I cannot say anything to you that was not said better in the movie itself" , which I love.

After a play or a good movie I'm digesting all my own feelings and reactions and I kind of just want to have private time to think about it, or maybe talk about it with one other sympatico soul. I don't want the writer putting the "official" way of thinking about it on me. And I hate when people know all kinds of things about movies or books that aren't in the material, like "oh the writer actually wrote this about his first marriage and his wife cheated on him" , fuck I don't care about the writer's real life backstory or who these characters really are, all it does is color my appreciation of the work in a negative way and distracts from the art. Leave the work alone.

Even without that aspects, Q&A sessions are balls. This guy doing the answering was actually really awesome, he has a quality that I much respect and admire, which is he could take really shitty questions and turn them into good answers. Dumb annoying nitty nerdy people will take stupid questions and just answer them completely factually and directly, which gives you dumb answers. That's not what the questioner wanted. As the question answerer, you have to ask yourself not "what did they say" but "what did they want to say" and "what do they want me to reply". Often you can guess what they might have been getting at (or *should* have been getting at) even if they failed to express it in their question.

But the problem is the people who ask the questions are the most annoying attention-hungry tards of all. We literally had people who would raise their hand and then say "I don't have a question but a comment .." and then proceed to give their own opinion about the play. What? Who the fuck are you? I don't want your opinion! It reminded me painfully of college where the only people to ever raise their hands are the fucking "is this on the test" people or the fucking "this is like what happened to me when I went to Cabo last year and ...".

Baguette Box is really damn good, but I wish they didn't "reimagine" the Banh Mi quite so imaginatively. Banh Mis are fucking great, most of the ways you've changed it are bad. Give me my pickled veg and chillis! And take the cilantro off the fucking stalk, it's inedible like that.

The frozen indian chickpeas at Trader Joes are the fucking bomb. It might be the best frozen food I've ever had. (I've always been a big fan of their pot pie).

The life of my laptop battery has been steadily declining over the years. It only lasts about an hour now. I guess I could replace it, but it's probably a better idea to just get a new laptop.

07-02-09 | Stores are Dicks

I'm fucking trying to give you my money, why are you being such a dick to me?

Larry wrote about some Amazon spam yesterday. He writes about the bit that says "don't contact us" claiming that it's a spammers trick. He is wrong about that. In fact that text is quite common from web sites now. For example I just bought some stuff from Cambria Bike and they send me this with the order confirmation email :

Please Note: There is no need to re-send your request or call our customer
service department for status or tracking number, this will only delay our
response time to you. Rest assured, we are making every effort to process
and ship your order within 1 to 2 business days. We appreciate your
understanding and patience and do value your business.

Look, I know you don't want people pestering you over and over after they put in an order, but this "don't contact us, it will delay our response time to you" is fucking bullshit and is just being a dick.

I was trying to buy my gear from Performance Bike, but their fucking online order system is broken. After spending like 30 minutes picking out all my gear and setting up my cart, I try to check out and the order system eats its own ass. So I email customer support :

I'm trying to check out and it tells me this -

"An unexpected error occurred while validating the submitted address."

Can I submit my cart by email and then phone in payment information so I don't have to
do everything over the phone?

This is my cart :

[... many items copy pasted ...]

Their reply :

Dear Mr. Bloom,

We are aware of the problem and are working to correct this. You may
call us at 1-800-727-2453 to place an order by phone. I apologize for
the inconvenience.

Umm, okay, so THAT'S A FUCKING NO. Just say "no, we are not going to try to help you with what you want at all, you can either suck our dicks or go shop elsewhere". Okay, I shop elsewhere.

This is a paraphrase of one my comcast chats :

[I've been having connection drop outs for a long time and keep contacting them about it over and over]

me: okay, it's getting really annoying having to call through the main number every time I have to talk to tech support, can I please have a direct number for tech support?

comcast: yes sir, the number is 1-800-266-2278

me : umm, no, that's 1-800-comcast , that's just the main number

comcast : yes sir

me : that's not a direct number for tech support, will you give me a direct number?

comcast : that is the number for tech support

me : are you refusing to give me a direct number for tech support ?

comcast : no sir, I did you give you the number


I would actually much prefer it they would just say "No we don't give out the direct number because we don't trust our customers".

(on the plus side I do seem to have working interent now, so that part's good).

07-01-09 | Public Transit

Should be free.

For one thing, it greatly reduces costs because you don't need all the ticket printing/selling/checking equipment and salaries for all those people who collect the money. You also don't need cops checking people's tickets. Because of this the cost for tax-supported free public transit would be much less than the total cost paid by ticket purchasers currently.

It also greatly improves efficiency because buses can just pick people up without the big stall for ticket buying and checking at the entry way. You can change your buses to have more doors and wider doors so that people can just flow on and off more easily and eliminate the huge pinch point that causes big slowdowns.

This will only appeal to liberals, but it's a nice way of giving back to the working poor. Most people who take the bus to work are working poor and can use every break they can get. Riding the bus will always be unpleasant so it's not like people will abuse the priviledge.

Now some people would argue that free public transit would overburden the system, there wouldn't be enough buses, they would get too crowded. That's completely retarded, that's exactly the outcome you *want*. You buy more buses!

I believe it may actually be cheaper in the long run. If you have more people on public transit you don't have to spend as much making new roads everywhere, people buy fewer cars saving them tons of money, we pollute less and have to spend less on reducing CO2, etc. there are tons of long term savings that are hard to quantify.

Now I'm sure the libertarian/republican/dickhead/ayn-randers out there are saying "but I don't ride the bus, why should I pay for it?" First of all, fuck you, you're a dick. Public policy should be optimized to maximize the total good (or perhaps to maximize the median good), the principle of "pay per use" does not enter into rational consideration of public policy at all. Second of all, *you* are the person who would get the most benefit from this. Personally I would probably keep driving because I hate the bus, but I would love to see free public transit, because it gets cars off the road and reduces traffic for me. It's win-win for everyone.

07-01-09 | Volunteer Park

Volunteer Park is an oasis of calm and beauty in this shitty pot-hole-ridden non-stopping-car-filled city.

I met Barefoot Ted in Volunteer Park the other day. Apparently he's running 45 miles barefoot around Green Lake for his 45th birthday today. He's like a barefoot jack lalanne.

The next day I discovered my PT Wolf is friends with Ted. Wolf sometimes wears Five Fingers shoes which are for barefoot runners who want protection. They look pretty sweet, I might pick up a pair just so I can enhance my goofball factor a bit.

The reservoir in the middle of Volunteer Park is calling to me. It's all fenced up because America is a bunch of cocks that locks everything and puts fences around everything fun. I've been fantasizing about getting a pair of bolt cutters and breaking in to swim around in it; it's a beautiful huge fresh water pool, it would be so sweet. Obviously I would get arrested, but it would be worth it.

Another act of civil disobedience I've been contemplating lately is painting crosswalks at all the intersections around here. It rankles me deeply the way the cars don't stop for pedestrians here, even on the quiet residential streets. I think it would be a simple pointed statement to go out in the night and paint crosswalks all over.

06-30-09 | Nissan 370Z

I finally went and drove a 370Z this morning. I skipped it in the earlier car testing because I think it's really ugly and I thought the G37 was just a better version of the same thing. I was wrong, the 370Z is fucking fantastic.

It feels small and tight and powerful, the steering feeling and response is excellent, the throttle is responsive. Obviously I didn't push it too hard on a test drive, but it just felt *fun* unlike any of the other cars I've driven. It's the lightest car I've driven at around 3200 pounds (still no featherweight by a longshot, but 300 less than the 135 and 400 less than the G37 and 800 less than the Audi RS4 or S5), and you can feel the difference.

There are some ridiculous things about it though. The seats suck, they're not very adjustable and they're just uncomfortable and cheap. If you lean your head back on the headrest, you can feel the lower inside bars of the headrest get levered forward and poke you in the back. The seats have sporty side flanges to pinch you, but they're not adjustable and are apparently made for very narrow people (the BMW's for example have power adjustable seat pinchers that lock you in very nicely).

The visibility is just absurd. Like dangerous. In fact I seriously think it should be illegal to make cars with such bad visibility. There's not really so much of a "blind spot" as you just can't look behind you *at all*. You have to use your side mirrors. The rear window is tiny, and the extreme angle of it means that you get bigtime fresnel effect so the glass is more reflective, and you actually can see the contents of your own trunk better than the cars behind you. Parallel parking in it would have to be done by closing your eyes and using The Force. Even ignoring the rearward visiblity problems, the front and side visibility isn't awesome either. Those windows are small too, and the hood and doors feel very high.

This is a common trend on lots of modern cars. The proper proportion for a car (says me) is about 50/50 body panel height to cabin glass height (more like 55/45 is ideal). Lots of modern cars are more like 65:35 or even 75:25. The hoods and doors are too high, you can't see the ground around you; it's claustrophobic and shitty. I think part of the reason is safety vs. the fucking big trucks and SUVs with their high bumpers, but I also think some people like the styling of it.

The 370Z also has lots of shitty plastic fit & finish. Whatever, I don't really care about that. The ergonomics of the console is actually better than BMW or Audi. The buttons are pretty simple and right at your hand and easy to use and well labelled. One thing that pisses me off in many news cars is fucking (+) and (-) buttons for things like air vent speed or stereo volume. That's fucking retarded. They need to be dials or wheels or knobs. I've ranted about this before, but look people - you want a few things from a control like that - you want the full range to be displayed physically, you want it to be easy to go instantly to any position, you want it to be friendly to muscle memory so that you can do it with your eyes away from it, and you need to be able to read the setting from the control. All of that is satisfied by the well known device of the knob (and I mean an absolute knob with the range and current value marked, not a digital knob that just spins forever in both directions). Stop fucking trying to reinvent the knob, you fail. Anyway, as cheap as the 370Z interior is, it actually has knobs FTW.

(while I'm ranting about nobs - fucking push button toggles are awful too, especially when the only indicator of what state it's in is from a dim LED that you can't tell if it's lit or not on a sunny day; switches should be actual physical switches, for all the same very obvious reasons - you can tell the state it's in by touching it with your eyes closed, and you get physical feedback so you know when you've switched it, and aside from all that it just feels cool. It's fucking awful that shit like the "M button" or the traction control buttons are little digital push button toggles; they should be big giant lever switches like something Nicola Tesla would switch so that when you are ready to tear it up you say "engage!" and flip a big fucking switch). Anyhoo, back on track...

It has a little more headroom than the G37, but less than the 135. I can just barely sit all the way up in the 370Z but it's certainly not comfortable to just sit in. The interior and trunk space is in general just ridiculously small, like you would have trouble getting a big load of groceries in it, and if you ever buy even the smallest piece of furniture you have to have it delivered.

So, I'm a little torn. It's the first car I've driven that got me hot and bothered when I floored it around a corner, but the bad seats and shit visibility are pretty big problems. It is $10k cheaper than a G37 or 135 so I guess that goes in the equation somewhere. Maybe I could spend that $10k getting the body panels replaced with a Jaguar E-Type body.

In general I'm disappointed with all these new cars. The engines in all of them are phenomenal, but all the other bits are stupidly messed up. In many ways they are all inferior to my old Prelude, which has delightful big windows and very simple functional ergonomically friendly dash controls, and fantastic raw steering feel, and everything is delightfully manual and non-computer-involved and simple. After every one of these fancy car test drives, I've been pleasantly happy to get back in the Lude. Granted, after mashing the hungry throttle of a modern engine the Lude feels incredibly slow and impotent, but least I have a nice view and no fucking beeping while I take 10 seconds to accelerate to 60.

I suppose I shouldn't be surprised. In my dreams people would just take their product and only change it in ways that are definite improvements, and leave alone the things that work just fine, but of course they don't do that. It all makes me think of MS Office or Visual Studio or something. Yes, the newer version are fundamentally much better at their core, they have better engines and some features that I really want, but they also completely fucking change the user interface and move the menus around and rename things for no god damn reason, and add all kinds of bells and whistles that I don't want and just kludge it all up. Modern cars remind me of that.

A 370Z

Psyche! That's what it should look like.

This is what it does look like

06-30-09 | Yaaaarg

Still no internet at home and Comcast has remotely rebooted my modem three more times. They've used up all the minutes on my pay-go phone so I also have no phone. Now I have to wait around at home for a technician to come again.

I'm going from being on hold with Comcast and navigating the infuriating automated menu (and my usual trick of just slamming on buttons to get to an operator doesn't work in their system), to sitting tense at home as upstairs neighbor stomps around on my head, to disputing health care bills or going to another fucking doctor or PT appointment, to commuting in the goddamn fucking traffic. I want to scream and/or punch someone. I am a ball of sadness and rage.

I've been a total dick to the people I talk to at Comcast; I don't mean to be, but the fucking automated menu system is so fucking awful, I'm in a complete rage by the time I get through to someone. Some of the amusing things it's done :

Navigate through the menus to the choice for "problems with internet". I finally get there and it just plays a recording "for customer support please dial XXXX" and hangs up. Of course that number was the one I had dialed. Yum.

When you actually get through to the right place for problems it plays a recording "most problems can be solved by rebooting your modem, we will do so now, then play music for 30 seconds while it powers up" WHAT !? NO !? STOP!! And you can't hit any buttons to escape out of that - and in fact if you mash on buttons trying to get past it, one of the buttons apparently is "please reboot me again" and it starts over that cycle.

I remember back in SLO when I had a similar kind of problem with my cable modem, it took like a week of talking to frustrating morons before I finally got escalated to a "stage 2 technician" (or was it a third stage guild navigator?) and then it was fixed in one day. In SLO it turns out the problem was that some backbone IP conflict that was spoofing my cable modem's public address (or something, I don't really know shit about the net). The point is that it had nothing to do with the wires and wasn't anything anybody needed to come to my house for. All it took was an actual computer guy to look into the cable-side networking problem.

(ADDENDUM : someone from that "comcast cares" email address mailed me back within about 6 hours. I have a technician coming out today theoretically so we'll see what they say; of course the connection has been fine so far today. Everyone I've actually gotten to talk to at Comcast has been pretty nice and reasonable, it's just so aggravating how hard it is to get through the system to talk to someone).

On the phone with HealthNet you get 60 seconds of "you can also view your claims and benefits online" FUCK YOU I KNOW THAT. Fortunately with HealthNet the system of frantically mashing buttons does get you through to a human.

HealthNet has been really horrible, I recommend against them as I recommend against Comcast. They have a policy of approving providers for PPO on an individual basis, which means when you go into an office which is "in network" you have to check every single individual person. Somehow over and over I keep getting treated by the one person in an office who's not "preferred". I'm not sure if the fucking providers are doing this on purpose (MTI and Olympic PT, you fuckers), or if it's just bad luck. It is a financial boon for the providers to fuck you in this way because it removes the contractual limit on charges. Early on I made the mistake of trusting that when I went to an office and they got my health insurance info in advance and told me it was fine and I would be covered that it meant they would give me to a provider that was preferred. LOL.

I'm also frustrated and annoyed with my lack of progress on Oodle. At times I feel like I'm writing some really good code, and lots of it, but when I step back and look at what I've got done in the last 8 months, I'm not happy with where I am. I'm literally doing nothing but working, I basically have no friends (that I see), no hobbies, I never go out or do anything but work and take care of fucking errands and todos and desperately try to sleep. And yet I feel like I'm not working enough. Part of the problem is definitely all the fucking PT which is such a huge time sink and distraction. I'm tempted to just say fuck it and give up on my body because it's a frustrating annoyance, and the stress of it all is half undoing any progress I make.

In the voice of Mark from Peep Show : "that's what I need, to sink my teeth into a double helping of work".

06-29-09 | Blurg

My internet has been out at home for four days now. Arg Comcast. I've spend probably an hour on the phone with them, partly because they refuse to give me a direct number to tech support. They're so evasive about it. I ask for the number for tech support and they say "okay" and then give me the number for general customer support. I say "umm no, that's the main number for customer service, I want the number for tech support" and they say "that is the number for tech support" ; one time I said "are you refusing to give me the number to tech support" and the guy was like "no, I have given you the number for tech support". Yeesh.

This Cable Modem Troubleshooting Tips is pretty good. With a Surfboard you can go to and see your own status and event logs. Of course good luck getting to talk to tech support person who understands any of that. The new thing at Comcast is apparently rebooting your modem. They just love rebooting your modem (they can do it by remote control). Any time you call up, it's "I'm going to reboot your modem, please wait" which takes two minutes and does absolutely nothing. I try to explain that I've done that myself many times to no avail.

Last Friday I was stuck in the elevator at RAD for about an hour. I got in with Mike and pressed the button and it took us up as usual, and then just stopped. None of the buttons did anything, it refused to budge. Turns out it stopped just a few inches away from the floor, and because it wasn't lined up it refused to open the doors. We waiting about an hour for the technicians to show up and pry the doors open. One of the weirdest things about the whole experience to me was that when the technicians showed up they didn't say "hello" or anything at all, there was no "hello, is everyone okay in there?" or "we're going to pry the door open now" - nothing. They just started banging on the door, then one of them climbed through the roof and jumped onto the top of the elevator with a huge BANG. I was like WTF, warn us that you're about to jump onto the fucking top of the elevator. It was weird.

Game Angst has a pretty good article about the problems of deferred shading that nobody talks about. I'm a fan of deferred shading in theory, but he has a good point. There are other problems - for example it makes the pipeline for alpha vs. non-alpha completely different (presumably you would render alpha stuff onto your scene using forward shading after the deferred shading is done). Also, it basically means you have to use the same shader on everything, or that you can't use many different types of shaders that take different parameters. Like if you wanted your main characters to be rendered with very nice spherical harmonic lighting and the rest of your stuff to be lit with N*L , you can't really do that with deferred shading. (obviously the uniformity and unification is one of the appealing things about deferred shading). Another one that popped into my head while reading that is the hacky semi-Lambertian falloff. Instead of just doing Clamp[ N*L ] , it's nice to do Clamp[ (N*L + C)/(1 + C) ] , which serves to let the light go "around the edge" a little. By giving the artists control of C per object you get a very cheap way to improve lighting, but with deferred shading you'd have to add an extra attribute channel which is absurd.

I keep looking at apartments. I had to pay July rent, so now I have a lot of time, so I thought I'd go ahead and try the "Secretary Solution". See the Bruss article "The art of a right decision" . The basic idea is to look at places for a while and not take any of them. Then when you hit a certain preset time, you switch modes and then take the first place which is the best seen so far. As I noted before, this maximizes the chance of getting the best place, but it doesn't maximize average real EV and when it fails it can be arbitrarily bad. Anyway, I plan to look without taking for another week or so, and then switch modes.

I looked at this Grand Apartment in 1903 Mansion . It'd fucking amazing. The kitchen and the floor plan are perfect - it's almost exactly what I would design myself, which is saying a lot. The kitchen is huge and open, but not too open - it's got counters and islands separating it from the living room (I hate places where the kitchen directly spills into the living room with no barrier). The big problem with this place is the location. It's on a really shitty street on the west side of Broadway (the east side of Broadway is the good part). The street has a bunch of halfway houses for men on parole; those are actually the good neighbors - the halfway houses have curfews and the guys are very quiet because they have rules and they don't want to get in trouble. The bad neighbors are all the other shitty apartments full of broke punk kids. Anyway, it's another place like the Duplex that kind of presents a quandary for me - it's fucking gorgeous inside this apartment, but not so hot outside, I dunno how to weight that.

There are also tons of houses coming for rent. So far none of them is quite ideal, but it's encouraging to see a bunch of them on the market. Most of the best places are still trying to sell, the neighborhood is just blanketed in "for sale" signs. Upstairs neighbor stimpy dickhead has been waking me up at 7 AM recently. I desperately need to get the fuck out of here. I wouldn't mind waking up early except that it puts my commute right in the middle of rush hour, which I usually try to avoid.

In other apartment news : padmapper is a much better version of the "craigslist on google maps" than the original "housingmaps.com". On padmapper you can hit the "+" to expand the filters and get much nicer control. PadMapper desperately needs to be able to save searches though, it's semi-useless without that.

Mercer Island around Mercer Way is a lovely place to ride. Getting there is miserable. It would be kinda sweet to live there, so I could hop right out my door and ride the loop. Of course I only like riding up here maybe 4 months out of the year since I don't do cold or wet. And if I really want nice riding out of my door I should move back to SLO where the riding is sweet and the weather is fair every day. Anyway, while I was riding I saw a cyclist getting carried to an ambulance on a stretcher. It was a scary reminder of the extreme danger of this hobby. On the plus side, I'm over my fear of speed that I've had since my last bad crash. I can now open up in fast descents and my body doesn't tighten up. I am still consciously choosing to be more careful and go slower, but it's a logical choice, not a panic reaction, so I'm happy about that. Two of my car crashes and one of my bike crashes were caused just by bad road conditions; I'm much more aware now of how likely it is to go around a corner and find a huge patch of oil or sand or something you can't control or avoid that's going to make you crash. People who speed around corners that they've never been around before are just retarded, it would be fine if it was only dangerous to yourself, but you never know when someone else is going to be there around that corner too. I consciously try to only speed around corners if I've been there before recently so I know the conditions. It would be really sweet to be able to ride on closed roads that are all smoothly paved and free of debris, but I guess you have to be a pro to get that. If I was billionaire I'd spend my money on shit like that. You could just rent the entire route you want to ride, country lanes in vermont, or mountain roads in the rockies, send a street sweeper ahead of you to make sure it's all clean, have any nasty bits repaved, and then have a beautiful ride. What would it cost, maybe $100k ? If you spend $100k every day for the rest of your life, it only reaches around $1 billion. No problem for a Gates or Buffet.

Seattle's summer days are quite wonderful. It's a shame to waste even a single one, since you only get maybe 20 total. You need to run around in the sun, have nothing in particular to do except enjoy the moments, sit on a patio and have drinks, play, breathe.

I need to have a child so that I have someone to play with. I just want to go the park and play tag and catch. I guess I could get a dog, but dogs are gross. I've always wanted a rent-a-dog service; I'd love to have a dog at the park, I just don't want one getting hair and slobber all over my house. It's kind of horrible to have a kid just to get someone to play with, but I suppose it's not as bad as the reasons why most people have kids - to "carry on the family name" (WTF is that) , to show the world how awesome you are by creating a child that's very successful (look at my son the doctor, aren't I such a great parent), or to make a little you that can do all the things you wish you'd done and have the breaks you didn't get so you can live vicariously through them.

The Vintage Seattle blog is super awesome. If you like old photos of cities (as I do) it's a treasure trove; go back through the old posts. It always blows my mind just how empty this country was 100 years ago ( for example ). The picture of the great white fleet is stunning.

Sean has put up his game Succor . It's an extremely interesting & clever concept; I won't spoil it, go download and play.

It occured to me the other day that since I use an HTPC to watch TV now, I could plug in a gamepad controller and play some little games on my TV (like Mutant Storm). LOL yeah right. I don't want to crash or reboot my fucking TV machine, which playing games would certainly cause.

I test-drove a G37 again a few days ago. It's a good car, I don't blame anyone who gets one, but I won't. It's a tiny bit too small, I hit my head on the roof. Supposedly by the official "front headroom" measurement it's bigger than the 135, but the reality is not so; I have plenty of room in the 135 but not in G37. The other problem is it just feels impotent at low revs, and it's too big and heavy. They brag about the "smoothness" of the acceleration; I don't want to ride a wave of acceleration, I want to be punched in the gut. The car should be gentle when I'm soft on the pedal, but it should jerk me around mercilessly when I thrash it.

Anyway, it was a funny test drive. The salesman guy was a total stereotypical douchebag car salesman; he had a baggy suit on and told me the trunk fit his golf clubs just fine. He told me has a G37 himself and got it lowered 1.5" so that he can't get over any bumps. I was like "ah, sweet", playing along trying to encourage the douchiness. He told me it looks great with bigger rims, "put some 20's on it". Yeah. So he showed me the bluetooth functionality, which granted is very good - much better than BMW - and he mentioned you can easily turn it off if you're in the car with someone and you don't want your calls to show up on the screen. I was like "oh yeah, if I'm riding with my girlfriend and my mistress calls I don't want that showing up" and he immediately said "yeah, I always turn it off, I told my girlfriend the bluetooth doesn't work in my car". OMG super lol. Nice job super douchebag car salesman, you win life.

The advantage of the G37 over the 135 is that it is more direct and mechanical; it has a real LSD, the steering feels much more connected and responsive to me, the pedal to acceleration response feels more analog and mechanical. That's all good. Also, the computer in the Infinitis is totally superior. I've seen BMWs and Audis and Mercs and in all of them the computer is the fucking suck, I would rather not have it, it's a total mess to use. The Infiniti is a touch screen for the mother fucking win. Plus the navigation is very simple and intuitive, it's got arrow keys and "enter" on the steering wheel so that you can do many things without looking, and it has voice activation and it actually works (as in, I tried it on a car I've never used before and it recognized my selections correctly on the first try with no problems). I was extremely impressed, it's by far the best car computer I've ever seen. I still would probably rather not have it, but if you really like car computers, it's the one.

06-26-09 | Guide to Health Care

God damn I keep getting screwed by providers and the insurance company. Here are some tips :

1. Do your own research, find the specific doctor you want. This can be hard to do, but you should be able to find the local specialist in your problem. I recommend someone young and up & coming because they will actually care and be up on modern techniques. The crucial thing here is the difference between a good doctor and any old doctor is like night and day. Don't be afraid to just leave a doctor after one visit if you don't think you're getting the attention you need.

Specifically - if you have an injury where you get an MRI, you should expect a doctor to actually LOOK at the MRI. The first two doctors I went to literally didn't ever look at it. They order MRI's and make a ton of money off you and then just read the examiner's report. The MRI examiner is not an MD and not qualified to be making judgements on your condition. Furthermore, if you have something like a sports injury, if the doctor does not actually look at your body, eg have you take your clothes off and move around and show him the disfunction - you should walk right out of there.

2. Do your own research on how you will be billed and who will treat you. This is tricky and you cannot trust ANYONE on this matter. You will be told that the person who will treat you is "covered" by your insurance. That doesn't mean anything. They may be outside your "network" which will cause them to bill you at a very high rate. Assuming you are on an HMO/PPO plan of some kind you must find specific providers who are "preferred" or "in network". You must do your own research on this because no one else will.

Pursuant to that - it's not enough to know that a certain office is "in network" for you. You must call ahead and get the name of the exact person who will be treating you. Twice I've gone to PT offices that I was told were "in network" for me, and then the exact person I was assigned to was out of network. This can be rather tricky - when you go in to a doctor who's in network for you, he might send you into a room to get xrays, if the xray tech is out of network, boom you're fucked. The classic one that people get bit by this was is the anaesthesiologist. Often when people get surgery they find a surprise bill for $10,000 from the anaesthesiologist who's out of network for them even though the doctor/hospital/everything else is preferred.

You really have to be a huge asshole about this, you need to call ahead and get the names of everyone who is going to treat you and check on them. When you go in to the office, you need to be firm, any time someone walks in a room to treat you, you have to say "who is this" and if they're not a name you've checked you have to say no. I know this is ridiculous and often not possible in practice.

3. When you get bills, go over everything with a fine toothed comb. Providers will bill you directly for the balance not covered by insurance. Quite often they do this wrong/illegally either on purpose or by accident. Of course the health insurance often fails to reimburse correctly too, so check on that as well. If the doctor takes your health insurance, then they are not allowed to bill more than the negotiated rates; this amount is normally marked "not allowed" on your explanation of benefits. Often the doctors will go ahead and bill this to you one way or another; this is called "balance billing" and it's specifically illegal. It can be tricky to spot sometimes because they mix it in with allowed billing, so you have to do the math and see that all the billing amounts add up right.

When you get bills that don't look right to you, you have to call your health insurance and your provider. If it's a question of "balance billing" you just confirm it with your health insurance, and then tell your provider to fuck off. The other common problem is something is getting billed at a higher rate or not covered when you think it should be. I've had more luck calling the provider about this kind of thing, the health insurance will just tell you tough luck. The provider can be a little funny about how exactly they bill things, so if they change the billing code and resubmit they may get more coverage; they are usually happy to do this, you may need to talk to the head of the billing department or whatever.

Anyhoo, if you have some kind of Orthopedic problem, I highly recommend Dr. Chris Wahl. I haven't actually had surgery from him, so I can't comment on his surgical skills (anecdotal reports on surgical outcomes are pretty worthless anyway, we need fucking public statistical analysis of doctor's outcomes, but of course the corrupt greedy bastards at the AMA would never allow that). He's the first doctor I've ever had that actually looked at my MRIs. He's also the first doctor who even gave me a proper physical visual exam, as in I took my shirt off and he immediately said "hmm your right shoulder is dropped, looks like you had a type 2 sepration". Yes! yes I did, and both my previous doctors failed to diagnose it.

06-22-09 | Redraw Dilemma

This apartment searching is really annoying me. I can't handle having "many balls in the air" ; when I put something on my todo list, I like to work at it until it's gone. God I fucking hate shit on my todo list (the fucking health care keeps reinserting itself on my todo list and it's pissing me off; they got me again today with some billing fuckup, but I digress...).

Anyway, it's reminding me of a concept I often think about. I'll call it "the redrawer's dilemma" but there must be a better/standard name for this.

The hypothetical game goes something like this :

You are given a bag with 100 numbers in it. You know the numbers are in [0,1000] but don't know how many of each number there are in the bag. You start by drawing a random number from the bag.

At each turn of play, you can either keep your current number (in which case that is your final score), or you can put your current number back in the bag and draw again, but drawing again costs you -1 that will be subtracted from your final score.

How do you play this game optimally?

There are two things that are interesting to me about this game in real life. One is that humans almost always play it incredibly badly, and the second is that when you finally decide to stop redrawing you're almost always unhappy about it (unless you got super lucky and draw a 900+ number).

The two classic human player errors in this game are the "I just started drawing, I shouldn't stand yet" and the "I can't stop now, I already passed on something better than this". The "I just started drawing, I shouldn't stand yet" guy draws something like an 800 on one of his early draws. He thinks dang that's really good, but maybe this bag just has lots of high numbers in it, I just started drawing, I should put some time into it. Now of course that reasoning is based in correct logic - if you have reason to believe that your chance of drawing higher is good enough to merit the cost of continued looking, then yes, do so, but just drawing more because "it's early" makes no sense - the game is totally non temporal, the cost of continuing drawing doesn't go up over time. This often leads into the "I can't stop now, I already passed on something better than this" guy, who's mainly motivated by pride and shame - he doesn't want to admit to himself that he made a big mistake passing early when he got a high number, so he has to keep drawing until he gets something better. He might draw an 800, then a whole mess of single digit numbers and he's thinking "oh fuck I blew it" and then he draws a 400. At that point he should stand and quit redrawing, but he can't, so he draws again.

The thing is, even if he played correctly and just took the 400 after passing on the 800, he would be really unhappy about. And if the early termination guy played correctly and just got an early 800 and didn't draw, he would be unhappy too, because he'd always be wondering if he could've done better.

The other game theory / logical fallacy that plagues me in these kind of things is "I'm already spending X I may as well spend X". First I was looking for places around $1500, then I bumped it to $1700, then $1900. Now I'm looking at places for $2500 cuz fuck it they're nicer and I was looking at places for $2000 so it's only $500 more.

In other news, hotpads is actually a pretty cool apartment search site. It seems they are just scraping craigslist and maybe some other classifieds sites, so it's not like they have anything new, but the map interface and search features and such are solid. One thing is really annoying me about it though - the wheel zooming in the map is totally broken, I keep trying to wheel zoom and it sends the map off the never never land. Urg!

In more random news, I've really enjoyed the "Wallander" series on PBS ; the stories are pretty retarded/ridiculous, but I like the muddled contemplative pace of it, and the washed out monochrome color palette.

06-21-09 | Fast Exp & Log

So in an earlier post I wrote about approximation of log2 and Ryg commented with links to Robin Green's great GDC 2003 talk : part1 (pdf) and part2 (pdf) ( main page here ).

It's mostly solid, but in part 2 around page 40 he talks about "fastexp" and "bitlog" and my spidey senses got tingling. Either I don't understand, or he was just smoking crack through that section.

Let's look at "bitlog" first. Robin writes it very strangely. He writes :

A Mathematical Oddity: Bitlog
  A real mathematical oddity
  The integer log2 of a 16 bit integer
  Given an N-bit value, locate the leftmost nonzero bit.
  b = the bitwise position of this bit, where 0 = LSB.
  n = the NEXT three bits (ignoring the highest 1)

    bitlog(x) = 8x(b-1) + n

  Bitlog is exactly 8 times larger than log2(x)-1

Bitlog Example
 For example take the number 88
88 = 1011000
b = 6th bit
n = 011 = 3
bitlog(88) = 8*(6-1)+3
= 43
  (43/8)+1 = 6.375
  Log2(88) = 6.4594
  This relationship holds down to bitlog(8)

Okay, I just don't follow. He says it's "exact" but then shows an example where it's not exact. He also subtracts off 1 and then just adds it back on again. Why would you do this :

    bitlog(x) = 8x(b-1) + n

  Bitlog is exactly 8 times larger than log2(x)-1

When you could just say :

    bitlog(x) = 8xb + n

  Bitlog is exactly 8 times larger than log2(x)

??? Weird.

Furthermore this seems neither "exact" nor an "oddity". Obviously the position of the MSB is the integer part of the log2 of a number. As for the fractional part of the log2, this is not a particular good way to get it. Basically what's happening here is he takes the next 3 bits and uses them for linear interpolation to the next integer.

Written out verbosely :

x = int to get log2 of
b = the bitwise position of top bit, where 0 = LSB.

x >= (1 << b) && x < (2 << b)

fractional part :
f = (x - (1 << b)) / (1 << b)

f >= 0 && f < 1

x = 2^b * (1 + f)

correct log2(x) = b + log2(1+f)

approximate with b + f

note that "f" and "log2(1+f)" both go from 0 to 1, so it's exact at the endpoints
but wrong in the middle

So far as I can tell, Robin's method is actually like this :

uint32 bitlog_x8(uint32 val)
    if ( val <= 8 )
        static const uint32 c_table[9] = { (uint32)-1 , 0, 8, 13, 16, 19, 21, 22, 24 };
        return c_table[val];
        unsigned long index;
        _BitScanReverse(&index,(unsigned long)val);
        ASSERT( index >= 3 );
        uint32 bottom = (val >> (index - 3)) & 0x7;
        uint32 blog = (index << 3) | bottom;

        return blog;

where I've removed the weird offsets of 1 and this just returns log2 times 8. You need the check for val <= 8 because shifting by negative amounts is fucked.

But you might well ask - why only use 3 bits ? And in fact you're right, I see no reason to use only 3 bits. In fact we can do a fixed point up to 27 bits : (we need to save 5 bits at the top to store the max possible integer part of the log2)

float bitlogf(uint32 val)
    unsigned long index;
    _BitScanReverse(&index,(unsigned long)val);

    uint32 vv = (val << (27 - index)) + ((index-1) << 27);

    return vv * (1.f/134217728); // 134217728 = 2^27

what we've done here is find the pos of the MSB, shift val up so the MSB is at bit 27, then we add the index of the MSB (we subtract one because the MSB it self starts the counting at one in the 27th bit pos). This makes a fixed point value with 27 bits of fractional part, the bits below the MSB act as the fractional bits. We scale to return a float, but you could of course do this with any # of fixed point bits and return a fixed point int.

But of course this is exactly the same kind of thing done in an int-to-float so we could use that too :

float bitlogf2(float fval)
    FloatAnd32 fi;
    fi.f = fval;
    float vv = (float) (fi.i - (127 << 23));
    return vv * (1.f/8388608); // 8388608 = 2^23

which is a lot like what I wrote about before. The int-to-float does the exact same thing we did manually above, finding the MSB and making the log2 and fractional part.

One note - all of these versions are exact for the true powers of 2, and they err consistently low for all other values. If you want to minimize the maximum error, you should bias them.

The maximum error of ( log2( 1 + f) - f ) occurs at f = ( 1/ln(2) - 1 ) = 0.442695 ; that error is 0.08607132 , so the correct bias is half that error : 0.04303566

Backing up in Robin's talk we can now talk about "fastexp". "fastexp" is doing "e^x" by using the floating point format again, basically he's just sticking x into the exponent part to get the int-to-float to do the 2^x. To make it e^x instead of 2^x you just scale x by 1/ln(2) , and again we use the same trick as with bitlog : we can do exact integer powers of two, to get the values in between we use the fractional bits for linear interpolation. Robin's method seems sound, it is :

float fastexp(float x)
    int i = ftoi( x * 8.f );
    FloatAnd32 f;
    f.i = i * 1512775 + (127 << 23) - 524288;
    // 1512775 = (2^20)/ln(2)
    // 524288 = 0.5*(2^20)

    return f.f;

for 3 bits of fractional precision. (note that Robin says to bias with 0.7*(2^20) ; I don't know where he got that; I get minimum relative error with 0.5)).

Anyway, that's all fine, but once again we can ask - why just 3 bits? Why not use all the bits of x as fractional bits? And if we put the multiply by 1/ln(2) in the float math before we convert to ints, it would be more accurate.

What we get is :

float fastexp2(float x)
    // 12102203.16156f = (2^23)/ln(2)
    int i = ftoi( x * 12102203.16156f );
    FloatAnd32 f;
    f.i = i + (127 << 23) - 361007;
    // 361007 = (0.08607133/2)*(2^23)

    return f.f;

and indeed this is much much more accurate. (max_rel_err = 0.030280 instead of 0.153897 - about 5X better).

I guess Robin's fastexp is preferrable if you already have your "x" in a fixed point format with very few fractional bits (3 bits in that particular case, but it's good for <= 8 bits). The new method is preferred if you have "x" in floating point or if "x" is in fixed point with a lot of fractional bits (>= 16).


I found the Google Book where bitlog apparently comes from; it's Math toolkit for real-time programming By Jack W. Crenshaw ; so far as I can tell this book is absolute garbage and that section is full of nonsense and crack smoking.


it's obvious that log2 is something like :

x = 2^I * (1+f)

(I is an int, f is the mantissa)

log2(x) = I + log2(1+f)

log2(1+f) = f + f * (1-f) * C

We've been using log2(1+f) ~= f , but we know that's exact at the ends and wrong in the middle
so obvious we should add a term that humps in the middle.

If we solve for C we get :

C = ( log2(1+x) - x ) / x*(1-x)

Integrating on [0,1] gives C = 0.346573583

hence we can obviously do a better bitlog something like :

float bitlogf3(float fval)
    FloatAnd32 fi;
    fi.f = fval;
    float vv = (float) (fi.i - (127<<23));
    vv *= (1.f/8388608);
    //float frac = vv - ftoi(vv);
    fi.i = (fi.i & 0x7FFFFF) | (127<<23);
    float frac = fi.f - 1.f;
    const float C = 0.346573583f;
    return vv + C * frac * (1.f - frac);

06-21-09 | Blog consultation

So I'm thinking about renting this place ; it's kind of ridiculous, it's a 2 BR which I don't need, it's $1950 which is an outrageous amount to pay for rent (and they're overcharging for the current market conditions). Sticking with the negative, it's a duplex, and the bottom halve is inhabited by the owner. I'm somewhat worried about that, I don't like the idea of seeing the owner all the time, it makes me feel like I'm being watched and judged which is a very unpleasant feeling. Oh, and the driveway is shared and only wide enough for one car so you have to ask each other to move to get your cars in and out, that seems awful.

On the plus side, the kitchen is huge and full of light and has a real gas stove and fume hood. The other huge plus is that it has a big private deck on the roof. Those are my dreams, just to be able to cook and sit outside in the morning while I have my coffee. And it's a good location. The back yard private deck is so much better than even the balconies on big apartment buildings, because the balconies are all right next to each other.

Oh, the other drag is that the landlord is a handyman and does the repairs himself. That's such a huge fucking disadvantage. I've had that at three apartments now and it's been a huge disaster each time. They're always super slow, if you ask them to fix something it seems to be code for "please tear up my apartment and then leave your tools in it for a month and maybe stop by for an hour every other week". Then they act so smug and pleased with themselves when they actually fix something for you. You know, I'm sure a professional would've done a better job much faster, and it's just your duty to have things fixed, so stop acting like I should praise your amazing skills and thank you over and over. In reality *you* should be thanking *me* for letting you play your handyman dressup game on my time in my home.

I also looked at this place in the Trace Lofts. It's ridiculous, it makes me angry. For one thing, like so many of the new apartments, it's basically a shoebox with a hole cut in one end. It's long and skinny and only one skinny side has windows. It's also just a bit open space with no rooms, no closets. Okay, fine, it's a "loft", it's all urban and trendy and cool. But the whole fucking point of those big industrial loft conversions is that they SUCK so they're really cheap, so broke artists can afford them, which is what makes them cool. This place is expensive as balls and it's targetted at yuppies who want to play like they're urban cool artists and pay out the nose for nice fixtures but still get all the suckitude of not actually having rooms or doors or closets. So dumb. (the penthouses on top of the Trace Lofts seem amazing, the one available is in the old building part). (hell the penthouse is only $2250 , well well worth the $250 for the upgrade) (if you want to go to outrageous rents like that this looks nice too).

If I don't get the ridiculous duplex I might just rent a house. You can get whole houses now for around $1600/mo that are just slightly out of the main area here. There are several in the 19th & Prospect area for around that, which is a bit of a long walk to the happening area but still walkable, and there's even one at 16th and Harrison which is not far at all. If you rent a house, noone lives above you.

06-17-09 | Inverse Box Sampling - Part 2

Okay, in Part 1.5 I asked about the downsample that was the best inverse of bilinear upsampling. I have a solution that pleases me.

Sean reminded me that he tackled this before; I dunno if he has any notes about it on the public net, he can link them. His basic idea was to do a full solve for the entire down-sampled image. It's quite simple if you think about. Consider the case of 2X up & down sampling. The bilinear filter upsample will make a high res image where each pixel is a simple linear combo of 4 low res. You take the L2 error :

E = Sum[all high res pixel] ( Original - Upsampled ) ^2

For Sean's full solution approach, you set Upsampled = Bilinear_Upsample( X) , and just solve this for X without any assumption of how X is made from Original. For an N-pixel low res image you have 4N error terms, so it's plenty dense (you could also artificially regularize it more by starting with a low res image that's equal to the box down-sample, and then solve for the deltas from that, and add an extra "Tikhonov" regularization term that presumes small deltas - this would fix any degenerate cases).

I didn't do that. Instead I assumed that I want a discrete local linear filter and solved for what it should be.

A discrete local linear filter is just a bunch of coefficients. It must be symmetric, and it must sum to 1.0 to be mean-preserving (flat source should reproduce flat exactly). Hence it has the form {C2,C1,C0,C0,C1,C2} with C0+C1+C2 = 1/2. (this example has two free coefficients). Obviously the 1-wide case must be {0.5,0.5} , then you have {C1,0.5-C1,0.5-C1,C1} etc. as many taps as you want. You apply it horizontally and then vertically. (in general you could consider asymetric filters, but I assume H & V use the same coefficients).

A 1d application of the down-filter is like :

L_n = Sum[k] { C_k * [ H_(2*n-k) + H_(2*n+1+k) ] }

That is : Low pixel n = filter coefficients times High res samples centered at (2*n * 0.5) going out both directions.

Then the bilinear upsample is :

U_(2n) = (3/4) * L_n + (1/4) * L_(n-1)

U_(2n+1) = (3/4) * L_n + (1/4) * L_(n+1)

Again we just make a squared error term like the above :

E = Sum[n] ( H_n - U_n ) ^2

Substitute the form of L_n into U_n and expand so you just have a matrix equation in terms of H_n and C_k. Then do a solve for the C_k. You can do a least-squares solve here, or you can just directly solve it because there are generally few C's (the matrix is # of C's by # of pixels).

Here's how the error varies with number of free coefficients (zero free coefficients means a pure box downsample) :

r:\>bmputil mse lenag.256.bmp bilinear_down_up_0.bmp  rmse : 15.5437 psnr : 24.3339

r:\>bmputil mse lenag.256.bmp bilinear_down_up_1.bmp  rmse : 13.5138 psnr : 25.5494

r:\>bmputil mse lenag.256.bmp bilinear_down_up_2.bmp  rmse : 13.2124 psnr : 25.7454

r:\>bmputil mse lenag.256.bmp bilinear_down_up_3.bmp  rmse : 13.0839 psnr : 25.8302
you can see there's a big jump from 0 to 1 but then only gradually increasing quality after that (though it does keep getting better as it should).

Two or three free terms (which means a 6 or 8 tap filter) seems like the ideal width to me - wider than that and you're getting very nonlocal which means ringing and overfitting. Optimized on all my test images the best coefficients I get are :

// 8 taps :

static double c_downCoef[4] = { 1.31076, 0.02601875, -0.4001217, 0.06334295 };

// 6 taps :

static double c_downCoef[3] = { 1.25 , 0.125, - 0.375 };

(the 6-tap one was obviously so close to those perfect fractions that I just manually rounded it; I assume that if I solved this analytically that's what I would get. The 8-tap one is not so obvious to me what it would be).

Now, how do these static ones compare to doing the lsqr fit to make coefficients per image ? They're 99% of the benefit. For example :

// solve :
lena.512.bmp : doing solve exact on 3 x 524288
{ 1.342242526 , -0.028240414 , -0.456030369 , 0.142028257 }  // rmse : 10.042138

// static fit :
lena.512.bmp :  // rmse : 10.116388


// static fit :
clegg.bmp :  // rgb rmse : 50.168 , gray rmse : 40.506

// solve :
fitting : clegg.bmp : doing lsqr on 3 x 1432640 , c_lsqr_damping = 0.010000
{ 1.321164423 , 0.002458499 , -0.381711250 , 0.058088329 }  // rgb rmse : 50.128 , gray rmse : 40.472

So it seems to me this is in fact a very simple and high quality way to down-sample to make the best reproduction after bilinear upsampling.

I'm not even gonna touch the issue of the [0,255] range clamping or the fact that your low res image should actually be considered discrete, not continuous.

ADDENDUM : it just occured to me that you might do the bilinear 2X upsampling using offset-taps instead of centered taps. That is, centered taps reconstruct like :

+---+    +-+-+
|   |    | | |
|   | -> +-+-+
|   |    | | |
+---+    +-+-+

That is, the area of four high res pixels lies directly on one low res pixel. Offset taps do :

+---+     | |
|   |    -+-+-
|   | ->  | |
|   |    -+-+-
+---+     | |

that is, the center of a low res pixel corresponds directly to a high res pixel.

With centered taps, the bilinear upsample weights in 1d are always (3/4,1/4) then (1/4,3/4) , (so in 2d they are 9/16, etc.)

With offset taps, the weights in 1d are (1) (1/2,1/2) (1) etc... that is, one pixel is just copied and the tweeners are averages.

Offset taps have the advantage that they aren't so severely variance decreasing. Offset taps should use a single-center down-filter of the form :


(instead of {C2,C1,C0,C0,C1,C2} ).

My tests show single-center/offset up/down is usually slightly worse than symmetric/centered , and occasionally much better. On natural/smooth images (such as the entire Kodak set) it's slightly worse. Picking one at random :

symmetric :
kodim05.bmp : { 1.259980122 , 0.100375561 , -0.378468204 , 0.018112521 }   // rmse : 25.526521

offset :
kodim05.bmp : { 0.693510045 , 0.605009745 , -0.214854612 , -0.083665178 }  // rgb rmse : 26.034 

that pattern holds for all. However, on weird images it can be better, for example :

symmetric :
c:\src\testproj>Release\TestProj.exe t:\test_images\color\bragzone\clegg.bmp f
{ 1.321164423 , 0.002458499 , -0.381711250 , 0.058088329 }  // rgb rmse : 50.128 , gray rmse : 40.472

offset :
c:\src\testproj>Release\TestProj.exe t:\test_images\color\bragzone\clegg.bmp f
{ 0.705825115 , 0.561705835 , -0.267530949 }  // rgb rmse : 45.185 , gray rmse : 36.300

so ideally you would choose the best of the two. If you're decompressing in a pixel shader you need another parameter for whether to offset your sampling UV's by 0.5 of a pixel or not.

ADDENDUM : I got Humus working with a KLT color transform. You just do the matrix transform in the shader after fetching "YUV" (not really YUV any more). It helps on the bad cases, but still doesn't make it competitive. It's better just to go with DXT1 or DXT5-YCoCg in those cases. For example :

On a pure red & blue texture :

Humus YCoCg :

rmse : 11.4551 , psnr : 26.9848
ssim : 0.9529 , perc : 80.3841%

Humus KLT with forced Y = grey :

KLT : Singular values : 56.405628,92.022781,33.752548
 KLT : 0.577350,0.577350,0.577350
 KLT : -0.707352,0.000491,0.706861
 KLT : 0.407823,-0.816496,0.408673

rmse : 11.4021 , psnr : 27.0251
ssim : 0.9508 , perc : 79.9545%

Humus KLT  :

KLT : Singular values : 93.250313,63.979282,0.230347
 KLT : -0.550579,0.078413,0.831092
 KLT : -0.834783,-0.051675,-0.548149
 KLT : -0.000035,-0.995581,0.093909

rmse : 5.6564 , psnr : 33.1140
ssim : 0.9796 , perc : 87.1232%

(note the near perfect zero in the last singular value, as it should be)

DXT1 :

rmse : 3.0974 , psnr : 38.3450
ssim : 0.9866 , perc : 89.5777%

DXT5-YCoCg :

rmse : 2.8367 , psnr : 39.1084
ssim : 0.9828 , perc : 88.1917%

So, obviously a big help, but not enough to be competitive. Humus also craps out pretty bad on some images that have single pixel checkerboard patterns. (again, any downsampling format, such as JPEG, will fail on the same cases). Not really worth it to mess with the KLT, better just to support one of the other formats as a fallback.

One thing I'm not sure about is just how bad the two texture fetches is these days.

06-19-09 | Apartment Codewords

"Modern" = shitty faux-modernist from the 70's with aluminum windows and beige carpet and particle board kitchen cabinets.

"Amenities" = a closet off the lobby has a stairmaster in it.

"Turn of the century Classic" / "vintage" / etc = run down, peeling paint, cracked windows, original wood-burning stove, etc.

"Heat maintained for all resident's comfort" = heat not run at all ever so the landlord can make more money.

"Vibrant community" = Noisy white trash neighbors hang out right outside your window.

"Act fast / rare opportunity / don't wait!" = this has been listed for 6 months and nobody wants it, please please be the sucker who takes it.

"Proximity to downtown / near businesses" = way the fuck out in some neighborhood you've never heard of.

"Great location" = more often than not I'm finding this means it's located directly on I-5 ; "West facing" means the same thing.

"View view view" = everything about this place is so shitty we want you to focus on what's outside.

06-18-09 | Things annoying me while apartment searching

1. Craigslist is fucking awful, but it's all I've got. I can't filter in any meaningful way, hell I can't even select for just 1 bedrooms or by neighborhood in Seattle. So I have to manually poke through the listings myself (of course the search is horribly broken), and I can't mark ads I've already seen before, and people keep relisting the same property over and over.

2. Walking around my neighborhood, I see *tons* of stuff for rent that's not on the internet. It seems like every building around here has a vacancy now as the market is crashing. What the fuck am I supposed to do with that? Oh, yay, you put a "for rent" sign outside the building. How many bedrooms? what floor? what square feet? am I supposed to phone every single fucking building in the whole city !? my god.

3. Ridiculously amateurish and unprofessional people renting these things. Some people I call & email over and over and they never get back to me at all. Good job. Others put up apartment listings that are just woefully lacking in information. My god, fucking list the square footage at least.

4. Intentional lies & left out information in the ads. Of course lots of the people who leave out the square feet do it on purpose, like the place that say "spacious 1 bedroom" and then you email them and they tell you it's 550 square feet. The big thing people do here is neighborhood lies. Everything around claims to be "Capitol Hill" - no, fuckers, First Hill down in the hospitals is not Cap Hill, fucking Central District half way down Rainier is not cap hill, fucking Eastlake is not cap hill, hell I've seen places across the fucking bridge on Beacon Hill claiming to be "Cap Hill". Liars.

5. The ridiculous pretention that we're not in a huge real estate crash. All the ads are like "rare unit available - act now ! prestigious building!" uhh, hello, half the fucking city is vacant or on sale right now. You can quit with pretending that I should feel fortunate that you are being kind enough to try to rent to me.

In general I'm a little torn about whether to hurry up and get out of here (god knows I need to get out of here), vs. take my time and find a really great place or wait for the market to crash a bit more. My "boots on the ground" view of the market here is that it has completely crashed, people are moving out left and right, but the sellers/landlords have not yet come face to face with the reality, so sale prices are still high and rents are still high. Right now there's a glut of supply that's just not moving, tons of condos are sitting empty unsold. In the next 6 months or so there's going to be a price crash.

I'm finding I'm still a sucker for these charming old 1910-1920 buildings. They're just so beautiful and full of charm and character; every time I look at a brand new building it just feels like a boring box, like a hotel room, and it makes me feel claustrophobic and ill. I dunno, maybe I'm attracted to the high I get from lead and mold poisoning.

For example I noticed these new condos Lumen are going on auction. Looking at the pictures just makes me angry and sick. Some developer just slapped together a bunch of drywall and sheets of metal and calls it "modern" "sleek" "urban" and wants to sell each condo for $500k. I bet you could build those kind of units a few days each. One can easily understand why there are so many new condos like this popping up all over the city - if people are dumb enough to buy them, it's a HUGE windfall for the developer.

Some of the more interesting projects around here :

First Church on 15th near Group Health was going to be converted to condos. I'm sure the project is going to die now, but it's at least sort of interesting. Of course they just put super cheapo generic "modern" hotel box shit inside it, but at least you have the old church around you.

The new unfinished condos on Cal Anderson Park are going to auction. I've been looking at them for a while now, wondering why they've been sitting there for months 90% finished. Well, apparently the developer ran out of money. It's kind of an amazing location, looking right out on the park, though that could also be a bit of a negative because you have zero privacy and there are a lot of hobos and kids in that park.

Harvard and Highland is the big new project going up in the historic mansion district. The condos are huge (2000 sqft) and crazy expensive, it doesn't make any sense IMO, but the web page is cool because it has an interactive map of the neighborhood with info on all the great homes around there. It's one of the best pages I've ever seen on the local robber baron's mansions.

Unrelated but Edith Macefield’s army of tattoos is cool.

There's still a ton of new construction around here that isn't done yet. All the massive amounts of shitty old buildings are going to be in trouble. Among other things, the population density in urban seattle is *massively* expanding these days, with tons of big condo projects in cap hill & especially South Lake Union. The infrastructure does not exist to handle all these people and no thought or money is being put into designing the growth of the city in a manageable constructive way.

06-17-09 | DXTC More Followup

I finally came back to DXTC and implemented some of the new slightly different techniques. ( summary of my old posts )

See the : NVidia Article or NVTextureTools Wiki for details.

Briefly :

DXT1 = my DXT1 encoder with annealing. (version reported here is newer and has some more small improvements; the RMSE's are slightly better than last time). DXT1 is 4 bits per pixel (bpp)

Humus BC4BC5 = Convert to YCoCg, Put Y in a single-channel BC4 texture (BC4 = the alpha part of DXT5, it's 4 bpp). Put the CoCg in a two-channel BC5 texture - downsampled by 2X. BC5 is two BC4's stuck together; BC5 is 8 bpp, but since it's downsampled 2x, this is 2bpp per original pixel. The net is a 6 bpp format

DXT5 YCoCg = the method described by JMP and Ignacio. This is 8 bpp. I use arbitrary CoCg scale factors, not the limited ones as in the previously published work.

Here are the results in RMSE : (modified 6-19 with new better results for Humus from improved down filter)

name DXT1 Humus DXT5 YCoCg
kodim01.bmp 8.2669 3.9576 3.8355
kodim02.bmp 5.2826 2.7356 2.643
kodim03.bmp 4.644 2.3953 2.2021
kodim04.bmp 5.3889 2.5619 2.4477
kodim05.bmp 9.5739 4.6823 4.5595
kodim06.bmp 7.1053 3.4543 3.2344
kodim07.bmp 5.6257 2.6839 2.6484
kodim08.bmp 10.2165 5.0581 4.8709
kodim09.bmp 5.2142 2.519 2.4175
kodim10.bmp 5.1547 2.5453 2.3435
kodim11.bmp 6.615 3.1246 2.9944
kodim12.bmp 4.7184 2.2811 2.1411
kodim13.bmp 10.8009 5.2525 5.0037
kodim14.bmp 8.2739 3.9859 3.7621
kodim15.bmp 5.5388 2.8415 2.5636
kodim16.bmp 5.0153 2.3028 2.2064
kodim17.bmp 5.4883 2.7981 2.5511
kodim18.bmp 7.9809 4.0273 3.8166
kodim19.bmp 6.5602 3.2919 3.204
kodim20.bmp 5.3534 3.0838 2.6225
kodim21.bmp 7.0691 3.5069 3.2856
kodim22.bmp 6.3877 3.5222 3.0243
kodim23.bmp 4.8559 3.045 2.4027
kodim24.bmp 8.4261 5.046 3.8599
clegg.bmp 14.6539 23.5412 10.4535
FRYMIRE.bmp 6.0933 20.0976 5.806
LENA.bmp 7.0177 5.5442 4.5596
MONARCH.bmp 6.5516 3.2012 3.4715
PEPPERS.bmp 5.8596 4.4064 3.4824
SAIL.bmp 8.3467 3.7514 3.731
SERRANO.bmp 5.944 17.4141 3.9181
TULIPS.bmp 7.602 3.6793 4.119
lena512ggg.bmp 4.8137 2.0857 2.0857
lena512pink.bmp 4.5607 2.6387 2.3724
lena512pink0g.bmp 3.7297 3.8534 3.1756
linear_ramp1.BMP 1.3488 0.8626 1.1199
linear_ramp2.BMP 1.2843 0.7767 1.0679
orange_purple.BMP 2.8841 3.7019 1.9428
pink_green.BMP 3.1817 1.504 2.7461

And here are the results in SSIM :

Note this is an "RGB SSIM" computed by doing :

SSIM_RGB = ( SSIM_R * SSIM_G ^2 * SSIM_B ) ^ (1/4)

That is, G gets 2X the weight of R & B. The SSIM is computed at a scale of 6x6 blocks which I just randomly picked out of my ass.

I also convert the SSIM to a "percent similar". The number you see below is a percent - 100% means perfect, 0% means completely unrelated to the original (eg. random noise gets 0%). This percent is :

SSIM_Percent_Similar = 100.0 * ( 1 - acos( ssim ) * 2 / PI )

I do this because the normal "ssim" is like a dot product, and showing dot products is not a good linear way to show how different things are (this is the same reason I show RMSE instead of PSNR like other silly people). In particular, when two signals are very similar, the "ssim" gets very close to 0.9999 very quickly even though the differences are still pretty big. Almost any time you want to see how close two vectors are using a dot product, you should do an acos() and compare the angle.

name DXT1 Humus DXT5 YCoCg
kodim01.bmp 84.0851 92.6253 92.7779
kodim02.bmp 82.2029 91.7239 90.5396
kodim03.bmp 85.2678 92.9042 93.2512
kodim04.bmp 83.4914 92.5714 92.784
kodim05.bmp 83.6075 92.2779 92.4083
kodim06.bmp 85.0608 92.6674 93.2357
kodim07.bmp 85.3704 93.2551 93.5276
kodim08.bmp 84.5827 92.4303 92.7742
kodim09.bmp 84.7279 92.9912 93.5035
kodim10.bmp 84.6513 92.81 93.3999
kodim11.bmp 84.0329 92.5248 92.9252
kodim12.bmp 84.8558 92.8272 93.4733
kodim13.bmp 83.6149 92.2689 92.505
kodim14.bmp 82.6441 92.1501 92.1635
kodim15.bmp 83.693 92.0028 92.8509
kodim16.bmp 85.1286 93.162 93.6118
kodim17.bmp 85.1786 93.1788 93.623
kodim18.bmp 82.9817 92.1141 92.1309
kodim19.bmp 84.4756 92.7702 93.0441
kodim20.bmp 87.0549 90.5253 93.2088
kodim21.bmp 84.2549 92.2236 92.8971
kodim22.bmp 82.6497 91.0302 91.9512
kodim23.bmp 84.2834 92.4417 92.4611
kodim24.bmp 84.6571 92.3704 93.2055
clegg.bmp 77.4964 70.1533 83.8049
FRYMIRE.bmp 91.3294 72.2527 87.6232
LENA.bmp 77.1556 80.7912 85.2508
MONARCH.bmp 83.9282 92.5106 91.6676
PEPPERS.bmp 81.6011 88.7887 89.0931
SAIL.bmp 83.2359 92.4974 92.4144
SERRANO.bmp 89.095 75.7559 90.7327
TULIPS.bmp 81.5535 90.8302 89.6292
lena512ggg.bmp 86.6836 95.0063 95.0063
lena512pink.bmp 86.3701 92.1843 92.9524
lena512pink0g.bmp 89.9995 79.9461 84.3601
linear_ramp1.BMP 92.1629 94.9231 93.5861
linear_ramp2.BMP 92.8338 96.1397 94.335
orange_purple.BMP 89.0707 91.6372 92.1934
pink_green.BMP 87.4589 93.5702 88.4219

Conclusion :

DXT5 YCoCg and "Humus" are both significantly better than DXT1.

Note that DXT5-YCoCg and "Humus" encode the luma in exactly the same way. For gray images like "lena512ggg.bmp" you can see they produce identical results. The only difference is how the chroma is encoded - either a DXT1 block (+scale) at 4 bpp, or a downsampled 2X BC4 block at 2 bpp.

In RGB RMSE , DXT5-YCoCg is measurably better than Humus-BC4BC5 , but in SSIM they are are nearly identical. This is because almost all of the RMSE loss in Humus comes from the YCoCg lossy color conversion and the CoCg downsampling. The actual BC4BC5 compression is very near lossless. (as much as I hate DXT1, I really like BC4 - it's very easy to produce near optimal output, unlike DXT1 where you have to run a really fancy compressor to get good output). The CoCg loss hurts RMSE a lot, but doesn't hurt actual visual quality or SSIM much in most cases.

In fact on an important class of images, Humus actually does a lot better than DXT5-YCoCg. That class is simple smooth ramp images, which we use very often in the form of lightmaps. The test images at the bottom of the table (linear_ramp and pink_green) show this.

On a few images where the CoCg downsample kills you, Humus does very badly. It's bad on orangle_purple because that image is specifically designed to be primarily in Chroma not Luma ; same for lena512pink0g.bmp ; note that normal chroma downsampling compressors like JPEG have this same problem. You could in theory choose a different color space for these images and use a different reconstruction shader.

Since Humus is only 6 bpp, size is certainly not a reason to prefer DXT1 over it. However, it does require two texture fetches in the shader, which is a pretty big hit. (BTW the other nice thing about Humus is that it's already down-sampled in CoCg, so if you are using something like a custom JPEG in YCoCg space with downsampled CoCg - you can just directly transcode that into Humus BC4BC5, and there's no scaling up or down or color space changes in the realtime recompress). I think this is probably what will be in Oodle because I really can't get behind any other realtime recompress.

I also tried something else, which is DXT1 optimized for SSIM. The idea is to use a little bit of neighbor information. The thing is, in my crazy DXT1 encoder, I'm just trying various end points and measuring the quality of each choice. The normal thing to do it to just take the MSE vs the original, but of course you could do other error metrics.

One such error metric is to decompress the block you're working on into its context - decompress into a chunk of neighbors that have already been DXT1 compressed & decompressed as well. Then compare that block and its neighbors to the original image in that neighborhood. In my case I used 2 pixels around the block I was working on, making a total region of 8x8 pixels (with the 4x4 DXT1 block in the middle).

You then compare the 8x8 block to the original image and try to optimize that. If you just used MSE in this comparison, it would be the same as before, but you can use other things. For example, you could add a term that penalizes not changes in values, but changes in *slope*.

Another approach would be to take the DCT of the 8x8 block and the DCT of the 8x8 original. If you then just take the L2 difference in DCT domain, that's no different than the original method, because the DCT is unitary. But you can apply non-uniform quantizers at this step using the JPEG visual quantization weights.

The approach I used was to use SSIM (using a 4x4 SSIM block) on the 8x8 windows. This means you are checking the error not just on your block, but on how your block fits into the neighborhood.

For example if the original image is all flat color - you want the output to be all flat color. Just using MSE won't give you that, eg. MSE considers 4444 -> 3535 to be just as good as 4444 -> 5555 , but we know the latter is better.

This does in fact produce slightly better looking images - it hurts RMSE of course because you're no longer optimizing for RMSE.

06-17-09 | Inverse Box Sampling - Part 1.5

In the previous post we attacked the problem :

If you are given a low res signal L and a known down-sampler D() (in particlar, box down sampling), find an up sampler U() such that :

L = D ( U( L ) )

and U( L ) is as close as possible to the actual high res signal that L was made from (unknown).

I'm also interested in the opposite problem :

If you are given a high res signal H, and a known up-sampler U() (in particular, bilinear filtering), find a down sampler D() such that :

E = ( H - U( D( H ) ) )^2 is minized

This is a much more concrete and tractable problem. In particular in games/3d we know we are forced to use bilinear filtering as our up-sampler. If you use box down-sampling for D() as many people do, that's horrible, because bilinear filtering and box-downsampling are both interpolating and variance reducing. That, they both take noisey signals and force them towards gray. If you know that U() is going to be bilinear filtering, then you should use a D() that compensates for that. It's intuitively obvious that D should be something a bit like a sinc to bring in some neighbors with negative lobes to compensate for the blurring aspect of bilinear upsample, but what exactly I don't know yet.

(note that this is a different problem than making mips - in making mips you are actually going to be viewing the mip at a 1:1 resolution, it will not be upsampled back to the original resolution; you would use this if you were trying to substitute a lower res texture for a higher one).

I haven't tried my hand at solving this yet, maybe it's been done? Much like the previous problem, I'm surprised this isn't something well known and standard, but I haven't found anything on it.

06-16-09 | Inverse Box Sampling

A while ago I posed this problem to the world :

Say you are given the box-downsampled version of a signal (I may use "image" and "signal" interchangeably cuz I'm sloppy). Box-downsampled means groups of N values in the original have been replaced by the average in that group and then downsampled N:1. You wish to find an image which is the same resolution as the source and if box-downsampled by N, exactly reproduces the low resolution signal you were given. This high resolution image you produce should be "smooth" and close to the expected original signal.

Examples of this are say if you're given a low mip and you wish to create a higher mip such that downsampling again would exactly reproduce the low mip you were given. The particular case I mainly care about is if you are given the DC coefficients of a JPEG, which are the averages on 8x8 blocks, you wish to produce a high res image which has the exact same average on 8x8 blocks.

Obviously this is an under-constrained problem (for N > 1) because I haven't clearly spelled out "smooth" etc. There are an infinity of signals that when downsampled produce the same low resolution version. Ideally I'd like to have a way to upsample with a parameter for smoothness vs. ringing that I could play with. (if you're nitty, I can constrain the problem precisely : The correlation of the output image and the original source image should be maximized over the space of all real world source images (eg. for example over the space of all images that exist on the internet)).

Anyway, after trying a whole bunch of heuristic approaches which all failed (though Sean's iterative approach is actually pretty good), I found the mathemagical solution, and I thought it was interesting, so here we go.

First of all, let's get clear on what "box downsample" means in a form we can use in math.

You have an original signal f(t) . We're going to pretend it's continuous because it's easier.

To make the "box downsample" what you do is apply a convolution with a rectangle that's N wide. Since I'm treating t as continuous I'll just choose coordinates where N = 1. That is, "high res" pixels are 1/N apart in t, and "low res" pixels are 1 apart.

Convolution { f , g } (t) = Integral{ ds * f(s) * g(t - s) }

The convolution with rect gives you a smoothed signal, but it's still continuous. To get the samples of the low res image, you multiply this by "comb". comb is a sum of dirac delta functions at all the integer coordinates.

F(t) = Convolve{ rect , f(t) }

low res = comb * F(t)

low res = Sum[n] L_n * delta_n

Okay ? We now have a series of low res coefficients L_n just at the integers.

This is what is given to us in our problem. We wish to try to guess what "f" was - the original high res signal. Well, now that we've written is this way, it's obvious ! We just have to undo the comb filtering and undo the convolution with rect !

First to undo the comb filter - we know the answer to that. We are given discrete samples L_n and we wish to reproduce the smooth signal F that they came from. That's just Shannon sampling theorem reconstruction. The smooth reconstruction is made by just multiplying each sample by a sinc :

F(t) = Sum[n] L_n * sinc( t - n )

This is using the "normalized sinc" definition : sinc(x) = sin(pi x) / (pi x).

sinc(x) is 1.0 at x = 0 and 0.0 at all other integer x's and it oscillates around a lot.

So this F(t) is our reconstruction of the rect-filtered original - not the original. We need to undo the rect filter. To do that we rely on the Convolution Theorem : Convolution in Fourier domain is just multiplication. That is :

Fou{ Convolution { f , g } } = Fou{ f } * Fou{ g }

So in our case :

Fou{ F } = Fou{ Convolution { f , rect } } = Fou{ f } * Fou{ rect }

Fou{ f } = Fou{ F } / Fou{ rect }

Recall F(t) = sinc( t - n ) , so :

Fou{ f } = Sum[n] L_n * Fou{ sinc( t - n ) } / Fou{ rect }

Now we need some Fourier transform knowledge. The easiest way for me to find this stuff is just to do the integrals myself. Integrals are really fun and easy. I won't copy them here because it sucks in ASCII so I'll leave it as an exercise to the reader. You can easily figure out the Fourier translation principle :

Fou{ sinc( t - n ) } = e^(-2 pi i n v) * Fou{ sinc( t ) }

As well as the Fourier sinc / rect symmetry :

Fou{ rect(t) } = sinc( v )

Fou{ sinc(t) } = rect( v )

All that means for us :

Fou{ f } = Sum[n] L_n * e^(-2 pi i n v) * rect(v) / sinc(v)

So we have the Fourier transform of our signal and all that's left is to do the inverse transform !

f(t) = Sum[n] L_n * Fou^-1{ e^(-2 pi i n v) * rect(v) / sinc(v) }

because of course constants pull out of the integral. Again you can easily prove a Fourier translation principle : the e^(-2 pi i n v) term just acts to translate t by n, so we have :

f(t) = Sum[n] L_n * h(t - n)

h(t) = Fou^-1{ rect(v) / sinc(v) }

First of all, let's stop and see what we have here. h(t) is a function centered on zero and symmetric around zero - it's a reconstruction shape. Our final output signal, f(t), is just the original low res coefficients multiplied by this h(t) shape translated to each integer point n. That should make a lot of sense.

What is h exactly? Well, again we just go ahead and do the Fourier integral. The thing is, "rect" just acts to truncate the infinite range of the integral down to [-1/2, 1/2] , so :

h(t) = Integral[-1/2,1/2] { dv e^(2 pi i t v) / sinc(v) }

Since sinc is symmetric around zero, let's take the two halves of the range around zero and add them together :

h(t) = Integral[0,1/2] { dv ( e^(2 pi i t v) + e^(- 2 pi i t v) ) / sinc(v) }

h(t) = Integral[0,1/2] { dv 2 * cos ( 2 pi t v ) * pi * v / sin( pi v) }

(note we lost the c - sinc is now sin). Let's change variables to w = pi v :

h(t) = (2 / pi ) * Integral[ 0 , pi/2 ] { dw * w * cos( 2 t w ) / sin( w ) }

And.. we're stuck. This is an integral function; it's a pretty neat form, it sure smells like some kind of Bessel function or something like that, but I can't find this exact form in my math books. (if anyone knows what this is, help me out). (actually I think it's a type of elliptic integral).

One thing we can do with h(t) is prove that it is in fact exactly what we want. It has the box-unit property :

Integral[ N - 1/2 , N + 1/2 ] { h(t) dt } = 1.0 if N = 0 and 0.0 for all other integer N

That is, the 1.0 wide window box filter of h(t) centered on integers is exactly 1.0 on its own unit interval, and 0 on others. In other words, h(t) reconstructs its own DC perfectly and doesn't affect any others. (prove this by just going ahead and doing the integral; you should get sin( N * pi ) / (N * pi ) ).

While I can't find a way to simplify h(t) , I can just numerically integrate it. It looks like this :


You can see it sort of looks like sinc, but it isn't. The value at 0 is > 1. The height of the central peak vs. the side peaks is more extreme than sinc, the first negative lobes are deeper than sinc. It actually reminds me of the appearance of a wavelet.

Actually the value h(0) is exactly 4 G / pi = 1.166243... , where "G" is Catalan's constant.

Anyway, this is all very amusing and it actually "works" in the sense that if you blow up a low-res image using this h(t) basis shape, it does in fact make a high res image that is smooth and upon box-down sampling exactly reproduces the low-res original.

It is, however, not actually useful. For one thing, it's computationally ridiculous. Of course you would precompute the h(t) and store it in a table, but even then, the reach of h(t) is infinite, and it doesn't get small until very large t (beyond the edges of any real image), so in practice every output pixel must be a weighted sum from every single DC values in the low res image. Even without that problem, it's useless because it's just too ringy on real data. Looking at the shape above it should be obvious it will ring like crazy.

I believe these problems basically go back to the issue of using the ideal Shannon reconstruction when I did the step of "undoing the comb". By using the sinc to reproduce I doomed myself to non-local effect and ringing. The next obvious question is - can you do something other than sinc there? Why yes you can, though you have to be careful.

Say we go back to the very beginning and make this reconstruction :

F(t) = Sum[n] L_n * B( t - n )

We're making F(t) which is our reconstruction of the smooth box-filter of the original. Now B(t) is some reconstruction basis function (before we used sinc). In order to be a reconstruction, B(t) must be 1.0 at t = 0, and 0.0 at all other integer t. Okay.

If we run through the math with general B, we get :

again :

f(t) = Sum[n] L_n * h(t - n)

but with :

h(t) = Fou^-1{ Fou{ B } / sinc(v) }

For example :

If B(t) = "triangle" , then F(t) is just the linear interpolation of the L_n

Fou{ triangle } = sinc^2 ( v)

h(t) = Fou^-1{ sinc^2 ( v) / sinc(v) } = Fou^-1{ sinc } = rect(t)

Our basis functions are rects ! In fact this is the reconstruction where the L_n is just made a constant over each DC domain. In fact if you think about it that should be obvious. If you take the L_n and make them constant on each domain, then you run a rectangle convolution over that - as you slide the rectangle window along, you get linear interpolation, which is our F(t).

That's not useful, but maybe some other B(t) is. In particular I think the best line of approach is for B(t) to be some kind of windowed sinc. Perhaps a Guassian-windowed sinc. Any real world window I can think of leads to a Fourier transform of B(t) that's too complex to do analytically, which means our only approach to finding h is to do a double-numerical-integration which is rather a disastrous thing to do, even for precomputing a table.

So I guess that's the next step, though I think this whole approach is a practical dead end and is now just a scientific curiosity. I must say it was a lot of fun to actually bust out pencil and paper and do some math and real thinking. I really miss it.

06-15-09 | Blog Roll

It's time now for me to give a shout out to all the b-boys in the werld.

Adventures of a hungry girl
Beautiful Pixels
Birth of a Game
bouliiii's blog
Capitol Hill Triangle
cbloom rants
Cessu's blog
Culinary Fool
David Lebovitz
Diary of a Graphics Programmer
Diary Of An x264 Developer
Eat All About It
Game Rendering
garfield minus garfield
Graphic Rants
Graphics Runner
Graphics Size Coding
Gustavo Duarte
His Notes
I Get Your Fail
Ignacio Castaño
Industrial Arithmetic
John Ratcliff's Code Suppository
Lair Of The Multimedia Guru
Larry Osterman's WebLog
level of detail
Lightning Engine
Lost in the Triangles
Mark's Blog
Married To The Sea
More Seattle Stuff
My Green Paste, Inc.
not a beautiful or unique snowflake
NVIDIA Developer News
Pete Shirley's Graphics Blog
Pixels, Too Many..
Real-Time Rendering
realtimecollisiondetection.net - the blog
Ryan Ellis
Seattle Daily Photo
Some Assembly Required
stinkin' thinkin'
Stumbling Toward 'Awesomeness'
surly gourmand
Sutter's Mill
Thatcher's rants and musings
The Atom Project
The Big Picture
The Data Compression News Blog
The Ladybug Letter
The software rendering world
TomF's Tech Blog
Visual C++ Team Blog
Void Star: Ares Fall
What your mother never told you about graphics development
Wright Eats
Bartosz Milewski's Programming Cafe

autogen from the Google Reader xml output. I would post the code right here but HTML EATS MY FUCKING LESS THAN SIGNS and it's pissing me off. God damn you.

SAVED : Thanks Wouter for linking to htmlescape.net ; I might write a program to automatically do that to anything inside a PRE chunk when I upload the block.

int main(int argc,char *argv[])
    char * in = ReadWholeFile(argv[1]);
    while( in && *in )
        in = skipwhitespace(in);
        if ( stripresame(in,"<outline") )
            char * title = strextracttok(in,'"',&in);
            in = strstr(in,"htmlUrl");
            char * url = strextracttok(in,'"',&in);
            lprintf("<A HREF=\"%s\"> %s </A> <BR>\n",url,title);
        in = strnextline(in);

    return 0;

06-14-09 | Biking / Fuck you Seattle

God dammit, there's no good biking around here. The bike lanes are meager, and half of the so-called "bike routes" run you right down busy streets with hardly any space (I'm looking at you, Howell-Stewart junction). The roads are in a shameful state, full of pot holes and ruts that are literally rattling the bolts loose on my bike (and banging them is a huge hazard and hurts my joints like a motherfuck). The Lake Washington loop is tolerable, but there are tons of bits with ridiculously bad pavement or narrow/ no shoulder, as well as plenty of major hazards, lots of commuting traffic, and bad routing.

My original right shoulder injury in 2006 was a separation that became frozen and is now still bugging me in the form of a winging scapula and arthritic AC joint. That crash was caused by a pot hole in San Francisco. Fucking pot holes.

Also, the fucking mini traffic circles they've tossed around cap hill and 28th are fucking retarded. They don't function as real traffic circles because they're too small; a real traffic circle works because being "in the circle" is a seperate state. The big problem with them is that cars have to swing really wide to get around them, and the road isn't wide, so cars swing right into the path of pedestrians, and cut right into bicycles. It's fucking awful. Actually I hate all the "Yield" streets around here too since half of them are at blind intersections and lots of dumb fucks come barrelling through them at high speed. All of these residential intersections should have full 4-way stops and painted crosswalks. Hell, more cities just need streets that are ped/bike only. For example Pike between Broadway and 12th should just be closed to cars. It would be fantastic for local businesses.

Kirkland's got this lovely pool right by my work, so I go to check when the lap swim hours are ... none. I mean, they do have lap swim from 5:30-7 am, but that may as well not exist. Even if I wanted to get up that early, it's fucking cold and gross that early, I want to swim in the afternoon sun you fucks. WTF you have this fucking great pool and you just can't open it? Presumably this is the same problem as the fucking roads, that there's no damn taxes and the governments are fucking dumb. It's such stupid cost saving though; you've spent all the money to make this pool, you clean it and pump it, and then you only have it open 6 hours a day.

Anyway, Colman park just south of I90 is really cool. Not the part down on the lake, that's okay, but it's obvious, what with it's views or Rainier and whatnot. The cool part is up Lake Washington Blvd S toward 31st Ave. You get the best effect if you park down at the bottom and walk up the hill - it has cool winding paths and stairs and bridges, and then at the top there's this huge public vegetable+flower garden that's like a hidden garden surprise for the hardy souls that made it up the hill.

Biking is so fucking great. I went and did the Mercer Island loop this weekend; it's pretty nice once you're out there, though the bike path is damn annoying and riding over I90 is scary and not fun. I have vertigo and the high view down to water with just a railing next to me is nauseau inducing. (only did the ride over the Golden Gate Bridge once; I nearly had a heart attack; after that I always drove my bike across the bridge and parked and then rode north).

I passed two seperate Mercer Island residents who seemed to intentionally stand right in my way. They were just standing in the road in the bike lane, cars were coming so it's not like I had a ton of room, and they made no movement out of the way at all. Rich people are fucking cocks.

Some douchebag cyclist dropped a passive-aggressive bummer-bumb on me. He was riding ahead of me on the bike lane, I move to the left and pass him. As I'm passing he says "I'm on your right". Huh? Yeah you are. I came up behind you and saw you. Oh, I get it. That's a fucking dickweed way of saying you expected me to say "on your left" when I passed you. Fuck you. It took me a little while to figure out what an acrid like asshole he was and by that time I was well past (because in additional to being a passive aggressive holier than thou dick he was also fat and slow); if I'd realized it sooner I would've yelled something back at him. How dare you fucking bring that negative shit into my world when I'm out on my ride having my one fucking moment of pure joy and pleasure? Fuck you, I know the fucking rules of courtesy, I say "on your left" if I think there's any danger or if it's a tight spot, but I don't say it every damn time I pass every person, and that's an unreasonable expectation, and even if you do think I should you can fucking keep it to yourself.

Some random dude also drafted me for a mile or so. That's not cool. You don't just jump on the ass of someone you don't know without saying anything. To draft correctly you have to be mere inches from the person in front of you. It's great for efficiency, but it's also very dangerous if you aren't communicating, because if the lead person brakes, you have an instant crash. Sometimes I'll latch onto someone's wheel when they pass me, but I hang back far enough to be safe, or I say hey can we draft a while? This dude just put his nose in my butt and stuck there. Mild scowl.

I got this book : Bicycling the Backroads around Puget Sound at the library. It sucks & basically proves that the biking here blows. There's one or two good rides in it (one of them being the Mercer Island Loop). Then it's full of rides that are just bullshit. It's got a bunch of rides that go down highways that are totally not suitable for biking (like the 203 and the 169) ; it also literally has a ride that goes on I90. WTF. Oh and then it's got rides that go off road. Umm, hello, this is the road biking book, you can put the unpaved road rides in another book, thank you. There are a few rides that look interesting, but they're well the hell far away, like Enumclaw or Granite Falls kind of far away. What the hell I guess I'll go try one soon cuz I don't have anything else to do.

The NYT Travel this week featured biking around Provence . That's like my dream; that article is pretty worthless and the writer is not a real biker, but doing some real country touring around Europe, in the sun of Provence, Tuscany, Catalonia, seeing all the countryside at the pace of a bike (the bike is the perfect speed for seeing country; walking is too slow and driving is too fast), eating and drinking. I don't want to ride the fucking Tour de France routes, that's way too hard and not fun.

06-14-09 | Noise Torture

I'm literally surrounded by crazy annoying loud people on all sides. To the west is the young couple who just had a baby that cries constantly; I've been around many babies and never heard one cry and cry like this; plus their Russian mom has now moved in with them and is always barking out orders to kill dissenting journalists in high pitched Russian.

To the north (fortunately across the street) is a party house full of frat boys who are constantly screaming "wooo" about something or other. Whoah bro sportscenter is on! Open the windows and scream "wooo" !! They literally set their house on fire the other day, fire trucks came and sprayed it, some fire dudes hacked open a wall to get at some stray embers inside. The next night after the fire they had another big party.

To the south is some amateur indie/punk band (emphasis strongly on the "amateur"). Fortunately they are a few houses away, but the pounding of the drums travels far. Sadly for me, they seem to be very dilligent about practicing the same song over and over and over. Sadly for them it doesn't seem to be helping.

To the east are the fucking white trash who are sitting outside drinking bud light and talking five feet away from my window. Blurg. Houses around here are way the fuck too close together. It's might be worse than the traditional row house like you have in SF or back east; with the row house you have a solid wall between you and the neighbor. Here we have open space and windows, but only like five feet of distance.

And of course there's "stompy" the crack head upstairs neighbor who seems to pass the time by moving his furniture from one end of the building to the other.

I guess it's kind of a noisy neighborhood, but I couldn't tell that when I moved in, because there are plenty of very nice single family homes around, with yuppie parents and kids and high property values.

There should really be segregation. There should be "ghettos" for the people who want to be noisy and have parties and whatnot - fine, that's cool, just go live in the noisy ghetto with other people of your kind. Alternatively, people should have to sign up in advance for certain weekends when they're allowed to be noisy so I can just go out of town those weekends.

I gather that many places in Europe have the tradition of a local pub on each block, and the people who live there just go congregate in the pub and make their noise there, so it's not in anyone's house. If you're making a ruckus, you go down to the pub. That seems like a good system.

Anyway, I write this now because by some divine conspiracy, all my neighbors went out of town this weekend. Hippie smoker jabberers next door - gone! Upstairs stompy - gone! No band practices, and no huge block parties. It was sublime, I was free, at peace, I could sit and read or work (I did lots of work), cook, listen to music, and I felt alone and happy. My god. Sometimes I get into these funks in life where I just think that everything fucking sucks and everyone is a huge dick, and then it's like the clouds clear - you see a moment where in fact, things do not suck, and it's like a revelation - whoah this misery is not how it has to be.

06-13-09 | Integer division by constants

.. can be turned into multiplies and shifts as I'm sure you know. Often on x86 this is done most efficiently through use of the "mul high" capability (the fact that you get 64 bit multiply output for free and can just grab the top dword).

Sadly, C doesn't give you a way to do mul high, AND stupidly MS/Intel don't seen to provide any intrinsic to do it in a clean way. After much fiddling I've found that this usually works on MSVC :

uint32 Mul32High(uint32 a,uint32 b)
    return (uint32)( ((uint64) a * b) >>32);

but make sure to check your assembly. This should assemble to just a "mul" and "ret edx".

Now, for the actual divider lookup, there are lots of references out there on the net, but most of them are not terribly useful because 1) lots of them assume expensive multiplies, and 2) most of them are designed to work for the full range of arguments. Often you know that you only need to work on a limited range of one parameter, and you can find much simpler versions for limited ranges.

One excellent page is : Jones on Reciprocal Multiplication (he actually talks about the limited range simplifications, unlike the canonical references).

The best reference I've found is the AMD Optimization Guide. They have a big section about the theory, and also two programs "sdiv.exe" and "udiv.exe" that spit out code for you! Unfortunately it's really fucking hard to find on their web site. sdiv and udiv were shipped on AMD developer CD's and appear to have disappeared from the web. I've found one FTP Archive here . You can find the AMD PDF's on their site, as these links may break : PDF 1 , PDF 2 .

And actually this little CodeProject FindMulShift is not bad; it just does a brute force search for simple mul shift approximations for limited ranges of numerators.

But it's written in a not-useful way. You should just find the shift that gives you maximum range. So I did that, it took two seconds, and here's the result for you :

__forceinline uint32 IntegerDivideConstant(uint32 x,uint32 divisor)
    ASSERT( divisor > 0 && divisor <= 16 );
    if ( divisor == 1 )
        return x;
    else if ( divisor == 2 )
        return x >> 1;
    else if ( divisor == 3 )
        ASSERT( x < 98304 );  
        return ( x * 0x0000AAAB ) >> 17; 
    else if ( divisor == 4 )
        return x >> 2;
    else if ( divisor == 5 )
        ASSERT( x < 81920 );  
        return ( x * 0x0000CCCD ) >> 18; 
    else if ( divisor == 6 )
        ASSERT( x < 98304 );  
        return ( x * 0x0000AAAB ) >> 18; 
    else if ( divisor == 7 )
        ASSERT( x < 57344 );  
        return ( x * 0x00012493 ) >> 19; 
    else if ( divisor == 8 )
        return x >> 3;
    else if ( divisor == 9 )
        ASSERT( x < 73728 );  
        return ( x * 0x0000E38F ) >> 19; 
    else if ( divisor == 10 )
        ASSERT( x < 81920 );  
        return ( x * 0x0000CCCD ) >> 19; 
    else if ( divisor == 12 )
        ASSERT( x < 90112 );  
        return ( x * 0x0000BA2F ) >> 19; 
    else if ( divisor == 13 )
        ASSERT( x < 98304 );  
        return ( x * 0x0000AAAB ) >> 19; 
    else if ( divisor == 13 )
        ASSERT( x < 212992 );  
        return ( x * 0x00004EC5 ) >> 18; 
    else if ( divisor == 14 )
        ASSERT( x < 57344 );  
        return ( x * 0x00012493 ) >> 20; 
    else if ( divisor == 15 )
        ASSERT( x < 74909 );  
        return ( x * 0x00008889 ) >> 19; 
    else if ( divisor == 16 )
        return x >> 4;
        return x / divisor;

Note : an if/else tree is better than a switch() because we're branching on constants. This should all get removed by the compiler. Some compilers get confused by large switches, even on constants, and fail to remove them.

7 seems to be the worst number for all of these methods. It only works here up to 57344 (not quite 16 bits).

06-13-09 | A little mathemagic

I'm doing some fun math that I'll post in the next entry, but first some little random shit I've done along the way.

First of all this is just a handy tiny C++ functor dealy for doing numerical integration. I tried to be a little bit careful about floating point issues (note for example the alternative version of the "t" interpolation) but I'm sure someone with more skills can fix this. Obviously the accumulation into "sum" is not awesome for floats if you have a severely oscillating cancelling function (like if you try to integrate a high frequency cosine for example). I suppose the best way would be to use cascaded accumulators (an accumulator per exponent). Anyhoo here it is :

template < typename functor >
double Integrate( double lo, double hi, int steps, functor f )
    double sum = 0.0;
    double last_val = f(lo);
    double step_size = (hi - lo)/steps;
    for(int i=1;i <= steps;i++)
        //double t = lo + i * (hi - lo) / steps;
        double t = ( (steps - i) * lo + i * hi ) / steps;
        double cur_val = f(t);
        // trapezoid :
        double cur_int = (1.0/2.0) * (cur_val + last_val);
        // Simpson :
        // double mid_val = f(t + step_size * 0.5);
        //double cur_int = (1.0/6.0) * (cur_val + 4.0 * mid_val + last_val);
        sum += cur_int;
        last_val = cur_val;
    sum *= step_size;
    return sum;

BTW I think it might help to use steps = an exact power of two (like 2^16), since that is exactly representable in floats.

I also did something that's quite useless but kind of educational. Say you want a log2() and for some reason you don't want to call log(). (note that this is dumb because most platforms have fast native log or good libraries, but whatever, let's continue).

Obviously the floating point exponent is close, so if we factor our number :

X = 2^E * (1 + M)

log2(X) = log2( 2^E * (1 + M) )

log2(X) = log2( 2^E ) + log2(1 + M)

log2(X) = E + log2(1 + M)

log2(X) = E + ln(1 + M) / ln(2)

M in [0,1]

Now, obviously ln(1 + M) is the perfect kind of thing to do a series expansion on.

We know M is "small" so the obvious thing that a junior mathematician would do is the Taylor expansion :

ln(1+x) ~= x - x^2/2 + x^3/3 - x^4/4 + ...

that would be very wrong. There are a few reasons why. One is that our "x" (M) is not actually "small". M is equally likely to be anywhere in [0,1] , and for M -> 1 , the error of this expansion is huge.

More generally, Taylor series are just *NEVER* the right way to do functional approximation. They are very useful mathematical constructs for doing calculus and limits, but they should only be used for solving equations, not in computer science. Way too many people use them for function approximation which is NOT what they do.

If for some reason we want to use a Taylor-like expansion for ln() we can fix it. First of all, we can bring our x into range better.

if ( 1+x > 4/3 )
  E ++
  x = (1+x)/2 - 1;

if (1+x) is large, we divide it by two and compensate in the exponent. Now instead of having x in [0,1] we have x in [-1/3,1/3] which is better.

The next thing you can do is find the optimal last coefficient. That is :

ln(1+x) ~= x - x^2/2 + x^3/3 - x^4/4 + k5 * x^5

for a 5-term polynomial. For x in [-epsilon,epsilon] the optimal value for k5 is 1/5 , the Taylor expansion. But that's not the range of x we're using. We're using either [-1/3,0] or [0,1/3]. The easiest way to find a better k5 is to take the difference from a higher order Taylor :

delta = k5 * x^5 - ( x^5/5 - x^6/6 + x^7/7 )

Integrate delta^2 over [0,1/3] to find the L2 norm error, then take the different wrst k5 to minimize the error. What you get is :

x in [0,1/3] :

k5 = (1/5 - 11/(18*12))

x in [-1/3,0] :

k5 = (1/5 + 11/(18*12))

it's intuitive what's happening here; if you just truncate a Taylor series at some order, you're doing the wrong thing. The N-term Taylor series is not the best N-term approximation. What we've done here is found the average of what all the future terms add up to and put them in as a compensation. In particular in the ln case the terms swing back and forth positive,negative, and each one is smaller than the last, so when you truncate you are overshooting the last value, so you need to compensate down in the x > 0 case and up in the x < 0 case.

using k5 instead of 1/5 we improve the error by over 10X :
basic    : 1.61848e-008
improved : 1.31599e-009

The full code is :

float log2_improved( float X )
    ASSERT( X > 0.0 );
    // approximate log2 by getting the exponent from the float
    //  and then using the mantissa to do a taylor series
    // get the exponent :
    uint32 X_as_int = FLOAT_AS_INT(X);
    int iLogFloor = (X_as_int >> 23) - 127;

    // get the mantissa :
    FloatAnd32 fi;
    fi.i = (X_as_int & ( (1<<23)-1 ) ) | (127<<23);
    double frac = fi.f;
    ASSERT( frac >= 1.0 && frac < 2.0 );

    double k5;

    if ( frac > 4.0/3.0 )
        // (frac/2) is closer to 2.0 than frac is
        //  push the iLog up and our correction will now be negative
        // the branch here sucks but this is necessary for accuracy
        //  when frac is near 2.0, t is near 1.0 and the Taylor is totally invalid
        frac *= 0.5;
        k5 = (1/5.0 + 11.0/(18.0*12.0));
        k5 = (1/5.0 - 11.0/(18.0*12.0));
    // X = 2^iLogFloor * frac
    double t = frac - 1.0;

    ASSERT( t >= -(1.0/3.0) && t <= (1.0/3.0) );

    double lnFrac = t - t*t*0.5 + (t*t*t)*( (1.0/3.0) - t*(1.0/4.0) + t*t*k5 );
    float approx = (float)iLogFloor + float(lnFrac) * float(1.0 / LN2);

    return approx;

In any case, this is still not right. What we actually want is the best N-term approximation on a certain interval. There's no need to mess about, because that's a well defined thing to find.

You could go at it brute force, start with an arbitrary N-term polynomial and optimize the coefficients to minimize L2 error. But that would be silly because this has all been worked out by mathemagicians in the past. The answer is just the "Shifted Legendre Polynomials" . Legendre Polynomials are defined on [-1,1] ; you can shift them to any range [a,b] that you're working on. They are an orthonormal transform basis for functions on that interval.

The good thing about Legendre Polynomials is that the best coefficients for an N-term expansion in Legendre polynomials are just the Hilbert dot products (integrals) with the Legendre basis functions. Also, if you do the infinite-term expansion in the Legendre basis, then the best expansion in the first N polynomials is just the first N terms of the infinite term expansion (note that this was NOT true with Taylor series). (the same is true of Fourier or any orthonormal series; obviously it's not true for Taylor because Taylor series are not orthonormal - that's why I could correct for higher terms by adjusting the lower terms, because < x^5 * x^7 > is not zero). BTW another way to find the Legendre coefficients is to start with the Taylor series, and then do a least-squares best fit to compensate each lower coefficient for the ones that you dropped off.

(note the standard definition of Legendre polynomials makes them orthogonal but not orthonormal ; you have to correct for their norm as we show below :)

To approximate a function f(t) we just have to find the coefficients : C_n = < P_n * f > / < P_n * P_n >. For our function f = log2( 1 + x) , they are :


which you could find by exact integration but I got lazy and just did numerical integration. Now our errors are :

basic    : 1.62154e-008
improved : 1.31753e-009
legendre : 1.47386e-009

Note that the legendre error reported here is slightly higher than the "improved" error - that's because the Legendre version just uses the mantissa M directly on [0,1] - there's no branch test with 4/3 and exponent shift. If I did that for the Legendre version then I should do new fits for the ranges [-1/3,0] and [0,1/3] and it would be much much more accurate. Instead the Legendre version just does an unconditional 6-term fit and gets almost the same results.

Anyway, like I said don't actually use this for log2 , but these are good things to remember any time you do functional approximation.

06-06-09 | Old Bike Stems

Urg. I just spent the entire fucking day trying to track down a replacement stem for my moderately old (mid 90's) racing bike. My bike uses a quill stem, which is pretty rare now (everyone has gone to threadless). The few quill stems you can find are targetted mainly at cruisers / mountain bikes. To make matters worse, even if you find one it has to be a match on several different sizing criteria.

First I had to figure out the sizes of my old stem so I could match. Some of the Stem Measurement guides just made me go WTF. Fortunately I found some nice clear ones like : this and this . So, yeah you just measure the stem along the bar that goes from the steerer axis to the handlebar axis. Road bike stems have a 73 degree angle, which makes them horizontal, a 90 degree stem will point up from the headset. This also had me confused for a second. If you changed from a 73 to a 90, the handlebars will actually do different things when you turn. Road bike handlebars actually dip down when you turn, they sort of lead you into leaning in the right direction. Neat!

Then you have the issue of all the tube diameters. Fortunately Sheldon has that sorted . The steerer tube part is almost always the ISO 1" size (which is 22.2 mm ; I know it should be 25.4 mm, but the 1" refers to the outside of the head tube, while the 22.2 is the diameter of the stem that fits into that). The handlebar diameter is a bit of a bigger problem. My bars don't have any label on them so I had to measure. The common sizes are 25.4, 25.8 or 26.0 mm. Of course I don't have calipers, so the easiest way to measure a diameter is by measuring the circumference with a piece of paper. Hah! Good luck telling the difference between 25.8 or 26.0 with a ruler. Anyway I know I don't have 25.4 , and the 25.8 and 26.0 are considered interchangeable.

Now I finally know what I want. A 22.2 mm - 26.0 mm stem with 100 mm reach. Sadly, the vast majority of cool old quill stems look like this . They have one bolt and the handlebar clamping bit wraps all the way around - you can't just take the handlebars in and out the way you can with modern threadless stems that have face plates like this . What that means is I would have to take off my brakes/shifters and hoods, take off my bar tape, slide the bar ends through the hole and twist it around, then put everything back on. Umm, no thanks, I'll have a face plate please.

Okay, I so I need a 22.2 mm - 26.0 mm quill stem with 100 mm reach and a removable face plate. Okay, let's track one down. Well, they do exist. One popular model is the Salsa SUL which is highly recommended various places. It's been recalled due to catastrophic dangerous metal failure. Umm yikes. Hmmm, well lucky me with more searching I tracked down another. The Deda Murex. Umm.. reputed to fail to hold handlebars , very loose and flexes easily. Umm.. okay, I found another, the Cinelli Frog ! Urg, same problem as the Deda.

I thought I found the mother lode at two cool eBay Stores : New Old Stock (NOS) and Mario's that have lots of classic bike gear. I started looking at a 3TTT Motus. (BTW 3TTT is is the most fucking retarded name ever, it's like Mount Fuji or something, and people either write it as "TTT" or "3T" or "3TTT" which annoys the fuck out of my Google searching). So the 3TTT Motus ... many cases of catastrophic clamp failure, strongly advised against. Yay. I think the Mutant might be fine, but I can't find a 100mm mutant.

Finally I found one - the Profile H2O ; it's a very common cheapo stem, I ruled it out at first because I only saw 90 and 105 degree models for the mountain/casual market. Then by chance I discovered they do make a 74 degree, so here I am. A full day later and $1000 of my time down the drain, I now get a shitty down-market aluminum stem.

BTW this might be the worst web site design ever. Oh yes, I know what this text needs - rotating and wiggling to make the viewer sick! And it should take forever to settle down so you have to try to read it while it wiggles or just sit there for minutes. To whoever made this : be ashamed.

I have a much worse problem upcoming with my around-town bike. It's an old bike in the 28.5" size which doesn't exist any more (it's all 700 now - BTW don't take those measurements too literally, they often don't actually correspond to anything that's actually on the bike). There's a lot of stuff I should do to fix it, but the parts are rare and expensive. I suppose I should give it away and get a new one, but it is an old CroMoly lugged frame that hardly exists any more except from bespoke custom builders.

I quite enjoy working on bikes. It's so much simpler and cheaper than working on cars, you can get all the tools you need for $100, and it's very satisfying to fix something with your own hands, and it brings you closer to your machine, you get to know how all the bits really work. I've written this before.

Sadly Seattle is an awful place to bike, but that's another rant. I'm still going, cuz hey, I still love biking even though the pavement quality and traffic and bike routes and countryside accessibility all suck balls here. It's like sex with a condom; yeah it's fucking awful and if you can have real sex you should, but if all you can get is sex with a condom, then you still do it. (people who tell you "well then don't do it" when you complain about something are fucking morons).

06-01-09 | Computers and Bodies

Computers destroy bodies in ways you probably aren't even aware of. I'm finally getting weekly massage and weekly PT that's nominally because of my shoulder problems, but really we spend just as much time working on computer-related problems. It costs a fortune. I'll probably spend $10k this year on massage and PT and pilates and chiropractors, and it takes a lot of time, but it's completely worth it.

A lot of people think they're okay because they aren't currently in pain or having numbness or carpal tunnel problems or whatever (BTW carpal tunnel is severely over-diagnosed and is also very easy to avoid; basically it's not a significant problem unless you are a retard and just mouse in a horrible position and ignore all the warning signs your body is sending you). In reality, all of these therapies can't really do much for you once you are in pain and having problems - the best time to work is *before* you have problems to prevent them.

Let me describe what happens to you when you sit at a computer day after day :

1. Your muscles just all generally atrophy because you're not doing a damn thing. Note that even if you work out this can be a big problem because you tend to only work your big movers, so you can get yourself into a dangerous situation where you have over-developed big movers and under-developed structural/stabilizer muscles.

2. Your hips lock up and your hamstrings shorten from sitting with knees bent all the time. Most of you actually sit on your low back, not your sit bones, which puts pressure on the vertebrae and nerves in the low back which can lead to sciatic pain and other nerve impingement disorders of the low body.

3. Your back rounds forward; obviously this happens badly if you slouch, but it also affects most people who try to be good and sit on a physioball or something, because they get tired and start resting on their arms and leaning into the monitor and keyboard. The back rounding forward does a lot of things - it shortens the muscles on the front of the body (mainly the pec minor) and it over-lengthens the muscles on the back (mainly the trapezies and teres major). Permanently stretched or compressed muscles are crippled - they can't execute their movement in their power zone near neutral. Back rounding also puts lots of bad pressure on the vertebrae, it pinches discs on the anterior side. Nerve bundles run out of your vertebrae through little holes and they get squeezed which leads to pain and weakness.

4. Your shoulders roll forward and get weak; partly because of #3. This is mainly because your arms are forward all the time, never above your head or even just relaxed at your side. The weight of your arms pulls the shoulder forward off where it should be resting. The shoulder is a very elaborate and delicate contraption - it doesn't have a ball and socket, the humerus just sort of sits up against the side of the body and is held in place by the rotator cuff tendon-muscular complex. By rolling the shoulder forward, parts of it are stretched and other compressed, which leads to weakness, constriction of nerves and blood flow, and pure mechanical disfunction (because it's in the wrong place, you can't get the right leverage with the right muscles and your body winds up compensating in bad ways).

Most people who have computer-related numbness or weakness or arm pain are actually have nerve pinching due to shoulder problems, not carpal tunnel. The nerve bundles from C5-C7 run through the shoulder and down the arm; they run through very small spaces which is fine if your body geometry is correct, but when you sit at a computer and your body gets all deformed with your head way forward and your upper back kyphotic and your shoulders rounded forward, it screws up the passages that the nerves should run through.

A lot of computer users think they are okay because they are working out or whatever. Certainly that is a good thing and a huge help, however you have to be very careful about how you do it and what you do.

There's a general societal problem that our image of the ideal male body right now is focused on abs and pecs. That leads people to over-develop the anterior muscles. This is like poison for computer user's bodies, because you are already rounded forward and the anterior muscles are over-shortened. Doing a bunch of crunches and bench presses will just make this work and do nothing to develop the stabilizers that you need. In fact this kind of training can make you even more primed for injury because you're moving heavy weights around and doing extreme athletic things without good stabilizers and basic body geometry.

BTW I love that Wikipedia has an entry for Tramp Stamp .

05-27-09 | Optimal Local Bases

I've had this idea forever but didn't want to write about it because I wanted to try it, and I hate writing about things before I try them. Anyway, I'm probably never going to do it, so here it is :

It's obvious that we are now at a point where we could use the actual optimal KLT for 4x4 transforms. That is, for a given image, take all the 4x4 blocks and turn them into a 16-element vector. Do the PCA to find the 16 bases vectors that span that 16-element space optimally. Tack those together to make a 16x16 matrix. This is now your KLT transform for 4x4 blocks.

(in my terminology, the PCA is the solve to find the spanning axes of maximum variance of a set of vectors; the KLT is the orthonormal transform made by using the PCA basis vectors as the rows of a matrix).

BTW there's another option which is to do it seperably - a 4-tap optimal horizontal transform and a 4-tap optimal vertical transform. That would give you two 4x4 KLT matrices instead of one 16x16 , so it's a whole lot less work to do, but it doesn't capture any of the shape information in the 2d regions, so I conjecture you would lose almost all of the benefit. If you think about, there's not much else you can do in a 4-tap transform other than what the DCT or the Hadamard already does, which is basically {++++,++--,-++-,+-+-}.

Now, to transform your image you just take each 4x4 block, multiply it by the 16x16 KLT matrix, and away you go. You have to transmit the KLT matrix, which is a bit tricky. There would seem to be 256 coefficients, but in fact there are only 15+14+13.. = 16*15/2 = 120. This because you know the matrix is a rotation matrix, each row is normal - that's one constraint, and each row is perpendicular to all previous, so the first only has 15 free parameters, the second has 14, etc.

If you want to go one step crazier, you could do local adaptive fitting like glicbawls. For each 4x4 block that you want to send, take the blocks in the local neghborhood. Do the PCA to find the KLT, weighting each block by its proximity to the current. Use that optimal local KLT for the current block. The encoder and decoder perform the same optimization, so the basis vectors don't need to be transmitted. This solve will surely be dangerously under-conditioned, so you would need to use a regularizer that gives you the DCT in degenerate cases.

I conjecture that this would rock ass on weird images like "barb" that have very specific weird patterns that are repeated a lot, because a basis vector will be optimized that exactly matches those patterns. But there are still some problems with this method. In particular, 4x4 transforms are too small.

We'd like to go to 8x8, but we really can't. The first problem is that the computation complexity is like (size)^8 , so 8x8 is 256 X slower than 4x4 (a basis vector has (size^2) elements, there are (size^2) basis vectors, and a typical PCA is O(N^2)). Even if speed wasn't a problem though, it would still suck. If we had to transmit the KLT matrix, it would be 64*63/2 = 2016 coefficients to transmit - way too much overhead except on very large images. If we tried to local fit, the problem is there are too many coefficients to fit so we would be severely undertrained.

So our only hope is to use the 4x4 and hope we can fix it using something like the 2nd-pass Hadamard ala H264/JPEG-XR. That might work but it's an ugly heuristic addition to our "optimal" bases.

The interesting thing about this to me is that it's sort of the right way to do an image LZ thing, and it unifies transform coding and context/predict coding. The problem with image LZ is that the things you can match from are an overcomplete set - there are lots of different match locations that give you the exact same current pixel. What you want to do is consider all the possible match locations - merge up the ones that are very similar, but give those higher probability - hmmm.. that's just the PCA!

You can think of the optimal local bases as predictions from the context. The 1st basis is the one we predicted would have most of the energy - so first we send our dot product with that basis and hopefully it's mostly correct. Now we have some residual if that basis wasn't perfect, well the 2nd basis is what we predicted the residual would be, so now we dot with that and send that. You see, it's just like context modeling making a prediction. Furthermore when you do the PCA to build the optimal local KLT, the eigenvalues of the PCA step tell you how much confidence to have in the quality of your prediction - it tell you what probability model to use on each of the coefficients. In a highly predictable area, the 1st eigenvalue will be close to 100% of the energy, so you should code predicting the higher residuals to be zero strongly; in a highly random area, the eigenvalues of the PCA will be almost all the same, so you should expect to code multiple residuals.

05-26-09 | Some Study of DCT coefficients

First of all : The number of non-zero values in the lower-diagonal area of the 8x8 block after quantization to a reasonable/typical value.

(36 of the 64 values are considered to be "lower diagonal" in the bottom right area) The actual number :

avg : 3.18 : 
 0 : 37.01
 1 : 16.35
 2 :  9.37
 3 :  6.48
 4 :  5.04
 5 :  3.88
 6 :  3.22
 7 :  3.02
 8 :  2.49
 9 :  2.34
10 :  2.09

The interesting thing about this is that it has a very flat tail, much flatter than you might expect. For example, if the probability of a given coefficient being zero or nonzero was an indepedent random event, the distribution would be binomial; it peaks flatter and is then much faster to zero :

avg : 3.18 : 
 0 : 13.867587
 1 : 28.162718
 2 : 27.802496
 3 : 17.775123
 4 : 8.272518
 5 : 2.986681
 6 : 0.870503
 7 : 0.210458
 8 : 0.043037
 9 : 0.007553
10 : 0.001150

What this tells us is the probability of a given coefficient being zero is highly *not* idependent. They are strongly correlated - the more values that are on, the more likely it is that the next will be on. In fact we see that if there are 6 values on, it's almost equally likely there are 7, etc. , that is : P(n)/P(n-1) goes toward 1.0 as n gets larger.

Also, amusingly the first two ratios P(1)/P(0) and P(2)/P(1) are both very close to 0.5 in every image I'm tried (in 0.4 to 0.6 generally). What this means is it wouldn't be too awful just to code the # of values on with unary, at least for the first bit (you could use something like an Elias Gamma code which uses unary at first then adds more raw bits).

Now for pretty pictures. Everyone has seen graphics like this, showing the L2 energy of each coefficient in the DCT : (none of these pictures include the DC because it's weird and different)

This shows the percentage of the time the value is exactly zero :

Now for some more interesting stuff. This shows the percent of correlation to the block above the current one : (to the north) :

Note in particular the strong correlation of the first row.

The next one is the correlation to the block to the left (west) :

Finally the fun one. This one shows the correlation of each coefficient to the other 63 coefficients in the same block :

The self-correlation is 100% which makes it a white pixel obviously. Black means 0% correlation. This is absolute-value correlation in all cases (no signs). There are a lot of patterns that should be pretty obvious to the eye. Beware a bit in over-modeling on these patterns because they do change a bit from image to image, but the general trend stays the same.

And another one from a different image :

This one's from Lena. A few things I think are particularly interesting - in the upper left area, which is where most of the important energy is, the correlation is most strong diagonally. That is, you see these "X" shape patterns where the center pixel is correlated mainly to it's diagonal neighbors, not the one's directly adjacent to it.

05-26-09 | Storing Floating Points

I rambled a bit before about how to store floating point images . I think I have the awesome solution.

First of all there are two issues :

Goal 1. Putting the floating point data into a format that I can easily run through filters, deltas from previous, etc. Even if you're doing lossless storage your want to be able to delta from previous and have the deltas make sense and preserve the original values.

Goal 2. Quantizing in such a way that the bits are used where the precision is wanted. Obviously you want to be sending bits only for values that are actually possible in the floating point number.

Also to be clear before I start, the issue here is with data that's "true floating point". That is, it's not just data that happens to be in floating point but could be equally well converted to ints inside some [min,max] range. For example, the coordinates of most geometry in video games really isn't meant to be floating point, the extra precision near zero is not actually wanted. The classic example where you actually want floating point is for HDR images where you want a lot of range, and you actually want less absolutely precision for higher values. That is, the difference between 999991 and 999992 is not as important as the difference between 1 and 2.

Now, we are going to be some kind of lossy storage. The loss might be very small, but it's kind of silly to talk about storing floating points without talking about lossy storage, because you don't really intend to have 23 bits of mantissa or whatever. To be lossy, we want to just do a simple linear quantization, which means we want to transform into a space where the values have equal significance.

Using some kind of log-scale is the obvious approach. Taking the log transforms the value such that even size steps in log-space are even multipliers in original space. That's good, it's the kind of scaling we want. It means the step from 1 to 2 is the same as the step from 100 to 200. The only problem is that it doesn't match the original floating point representation really, it's more continuous than we need.

What we want is a transform that gives us even size steps within one exponent, and then when the exponent goes up one, the step size doubles. That makes each step of equal importance. So, the quantizer for each exponent should just be Q ~= 2^E.

But that's easy ! The mantissa of the floating point that we already have is already quantized like that. We can get exactly what we want by just pretending our floating point is fixed point !

value = (1 + M)* 2^E

(mantissa with implicit one bit at head, shifted by exponent)

fixed point = {E.M}

That is, just take the exponent's int value and stick it in the significant bits above the mantissa. The mantissa is already quantized for you with the right variable step size. Now you can further quantize to create more loss by right shifting (aka only keeping N bits of the mantissa) or by dividing by any number.

This representation also meets Goal 1 - it's now in a form that we can play with. Note that it's not the same as just taking the bytes of a floating point in memory - we actually have an integer now that we can do math with and it makes sense.

when you cross an exponent :

1.99 = {0.1111}
2.01 = {1.0001}

So you can just subtract those and you get a sensible answer. The steps in the exponent are the correct place value to compensate for the mantissa jumping down. It means we can do things like wavelets and linear predictors in this space.

Now there is a bit of a funny issue with negative numbers vs. negative exponents. First the general solution and what's wrong with it :

Negatives Solution 1 : allow both negative numbers and negative exponents. This creates a "mirrored across zero" precision spectrum. What you do is add E_max to E to make it always positive (just like the IEEE floating point), so the actual zero exponent is biased up. The spectrum of values now looks like :

  (positives)      real step size
  3.M                     /
  2.M                    / 
  1.M                   /
  0.M                  /
 -1.M                 /
 -2.M                /  
 -3.M               /  
(zero)             +    
 -3.M               \   
 -2.M                \  
 -1.M                 \ 
  0.M                  \
  1.M                   \
  2.M                    \
  3.M                     \

What we have is a huge band of values with very small exponents on each side of the zero. Now, if this is actually what you want, then fine, but I contend that it pretty much never is. The issue is, if you actually have positives and negatives, eg. you have values that cross zero, you didn't really intend to put half of your range between -1 and 1. In particular, the difference between 1 and 2^10 is the same as the difference between 1 and 2^-10. That just intuitively is obviously wrong. If you had a sequence of float values like :

{ 2.0 , 1.8 , 1.4, 1.0, 0.6, 0.2, -0.1, -0.3, -0.6, -1.0, -1.4 }

That looks nice and smooth and highly compressible right? NO ! Hidden in the middle there is a *MASSIVE* step from 0.2 to -0.1 ; that seems benign but it's actually a step past almost your entire floating point range. (BTW you might be thinking - just add a big value to get your original floating poin tdata away from zero. Well, if I did that it would shift where the precision was and throw away lots of bits; if it's okay to add a big value to get away from zero, then our data wasn't "true" floating point data to begin with and you should've just done a [min,max] scale instead).

So I contend that is almost always wrong.

Negatives Solution 2 : allow negative numbers and NOT negative exponents. I content that you almost never actually want negative exponents. If you do want precision below 1.0, you almost always just want some fixed amount of it - not more and more as you get smaller. That can be represented better just with the bits of the mantissa, or by scaling up your values by some fixed amount before transforming.

To forbid negative exponents we make everything in [0,1.0] into a kind of "denormal". We just give it a linear scale. That is, we make a slightly modified representation :

Stored val = {E.M} (fixed point)

Float val =
    if E >= 1 : (1 + M) *2^(E-1)
                makes numbers >= 1.0

    if E == 0 : M
                makes numbers in [0.0,1.0)

(M is always < 1, that is we pretend it has a decimal point all the way to the left)

Now the low values are like :

0    : 0.0000
0.5  : 0.1000
0.99 : 0.1111
1.01 : 1.0001

Of course we can do negative values with this by just putting the sign of the floating point value onto our fixed point value, and crossing zero is perfectly fine.

05-26-09 | Automatic Multithreading

Bartosz made an interesting post about extending D for automatic multithreading with some type system additions.

It made me think about how you could do guaranteed-safe multithreading in C++. I think it's actually pretty simple.

First of all, every variable must be owned by a specific "lock". It can only be accessed if the current thread owns that lock. Many of the ownership relationships could be implicit. For example there is an implicit lock for every thread for stuff that is exlusively owned by that thread. That thread almost has that lock, so you never actually generate a lock/unlock, but conceptually those variables still have a single lock that owns them.

So, stack variables for example are implicitly automatically owned by the thread they are made on. Global variables are implicitly owned by the "main thread" lock if not otherwise specified. If some other thread tries to touch a global, it tries to take a lock that it doesn't own and you fail.


Lock gate1;

int shared : gate1; // specifies "shared" is accessed by gate1

int global; // implicitly owned by the main thread

void thread1()
    int x; // owned by thread1

    x = shared; // fail! you must lock gate1

        x = shared; // ok

    y = global; // fail! you don't own global

Mkay that's nice and all. Single threaded programs still just work without any touches, everything is owned by the main thread. Another goal I think of threading syntax additions should be that going from a large single threaded code base to adding a few threading bits should be easy. It is here.

There are a few things you would need to make this really work. One is a clean transfer of ownership method as Bartosz talks about. Something like auto_ptr or unique_ptr, but actually working in the language, so that you can pass objects from one owner to another and ensure that no refs leak out during the passage.

You can also of course extend this if you don't want the constraint that everything is protected by a lock. For example you could easily add "atomic" as a qualifier instead of a lock owner. If something is marked atomic, then it can be accessed without taking a lock, but it's only allowed to be accessed by atomic operations like cmpx/CAS.

This is a nice start, but it doesn't prevent deadlocks and still requires a lot of manual markup.

I also finally read a bit about Sun's Rock. It's very interesting, I encourage you to read more about it at the Sun Scalable Synchronization page.

Rock is actually a lot like LRB in many ways. It's 16 lightweight cores, each of which has 2 hardware threads (they call them strands). It's basically a simple in-order core, but it can do a sort of state-save push/pop and speculative execution. They have cleverly multi-purposed that functionality for both the Transactional Memory, and also just for improving performance of single threaded code. The state-save push-pop is a ghetto form of out-of-order-execution that we all know and love so much. It means that the chip can execute past a branch or something and then if you go the other way on the branch, it pops the state back to the branch. This is just like checkpointing a transaction and then failing the commit !

The key thing for the transactional memory is that Rock is NOT a full hardware transactional memory chip. It provides optimistic hardware transactions with some well designed support to help software transactional memory implementations. The optimistic hardware transactions basically work by failing to commit if any other core touches a cache line you're writing. Thus if you do work in cache lines that you own, you can read data, write it out, it gets flushed out of the cache to the global consistent view and there are no problems. If someone touches that cache line it invalides the transaction - even though it might not necessarilly need to. That's what makes it optimistic and not fully correct (there are other things too). If it allows a transaction through, then it definitely was okay to do, but it can fail a transaction that didn't need to fail.

There's a lot of problems with that, it can fail in cases that are perfectly valid transactions, so obviously you cannot rely on that as a TM implementation. However, it does let a lot of simple transactions successfully complete quickly. In particular for simple transactions with no contention, the optimistic hardware transaction completes no problemo. If it fails, you need to have a fallback mechanism - in which case you should fall back to your real STM implementation, which should have forward progress guarantees. So one way of look at the HTM in Rock is just a common case optimization for your STM software. The commit in Rock has a "jump on fail" so that you can provide the failure handling code block; you could jump back to your checkpoint and try again, but eventually you have to do something else.

Perhaps more interestiongly, the HTM in Rock is useful in other ways too. It gives you a way to do a more general ll-sc (load linked store conditional) kind of thing, so even if you're not using STM, you can build up larger "atomic" operations for your traditional lock/atomic C++ style multithreading. It can also be used for "lock elision" - avoiding actually doing locks in traditional lock-based code. For example if you have code like :


    x = y;


that can be transparently converted to something like :

    checkpoint; // begin hardware transaction attempt

    x = y;

    commit  // try to end hardware transaction, if fails :
    failure {

        lock( CS)   
        x = y;


So you avoid stalling if it's not needed. There are of course tons of scary subtleties with all this stuff. I'll let the Sun guys work it out.

It's also actually a more efficient way of doing the simple "Interlocked" type atomic ops. On x86 or in a strongly ordered language (such as Java's volatiles) the Interlocked ops are fully sequentially consistent, which means they go in order against each other. That actually is a pretty hard operation. We think of the CMPX or CAS as being very light weight, but it has to sync the current core against all other cores to ensure ops are ordered. (I wrote about some of this stuff before in more detail in the threading articles). The "transaction" hardware with a checkpoint/commit lets your core flow through its ops without doing a big sync on interlocked ops. Now, obviously the checkpoint/commit itself needs to be synchronized against all other cores, but it's done on a per cache-line basis, and it uses the cache line dirtying communication hardware that we need anyway. In particular in the case of no contention or the case of doing multiple interlocked ops within one cache line, it's a big win.

I'm sure that Sun will completely cock things up somehow as they always do, but it is very interesting, and I imagine that most future chips will have models somewhat like this, as we all go forward and struggle with the issue of writing software for these massively parallel architectures.

05-25-09 | Undefined C

There's a lot of random little shit in C that's technically "undefined" (meaning the compiler/hardware are allowed to do anything they want). Basic stuff like right shifting signed values or doing addition that overflows or casting between different sizes/types of ints.

It's fucking retarded. It's a dangerous and pointless cop out. C is supposed to be the low-level systems language, it should have ways of clearly exposing the actual function of the system.

First of all, there should just be a "Normal C" machine spec that clearly specifies the behavior that 99% of current hardware has (like right shifting signed values shifts in sign bits). Then we could just say "this platform is a Normal C compliant platform".

But even aside from that, just saying "it's undefined" is a horrible way to support varying platforms. What you should do is have requirements and contracts as meta-code in the programs. A program might start out as only min-spec C. In that case, any use of undefined behavior should be a compile error! You tried to do something the platform does not offer, you fail compile!

Then, if the program needs certain things, it can add them to its requirements list, like "I need signed right shift to behave like this". Now that program will only compile on systems that provide that. The needs of the code are clearly listed and the contract is enforced. It should never be possible to access undefined behavior. The bad behavior should either be forbidden, or it should be enforced to do something specific.

All good robust software should be written this way. It's unforgiveable that our basic language tool isn't.

Instead we have a situation where someone can write code like :

    int64 a = ...;
    uint32 b = (uint32) ( a >> 16 );

WTF does that do? Does it work on this new platform I'm trying to compile on ?

Of course you/me as a user programmer can help the situation by putting in lots of in-line unit tests, like :

    int64 a = ...;
    uint32 b = (uint32) ( a >> 16 );

    UNIT_TEST( (uint32) ( ((int64)-1) >> 16 ) == 0xFFFFFFFF );

05-25-09 | Foodies

I fucking despise Michael Pollan and all the modern "foodies". However, I am occasionally quite grateful that I can partake of the purveyance that they have made possible. For example : Trader Joe's now carries Burrata (cream filled fresh mozarella balls). It's enough to make a lactard like myself endure the searing stomach pains. Also, I get to go to this crazy Mangalitsa Pig Feast .

I feel like I should explain and justify my hate a bit more.

Why my hate for Pollan - it's mainly his delivery. Everything he says is basically true, though he does overblow things rather a bit, but it's all really fucking obvious, it's like anybody with half a fucking brain knows that already, and people without a brain aren't actually learning, they're just blindly following the new preacher instead of the old one. The thing that tilts me is his condescending fucking holier than thou smirk, you can see the look on his face like he things he's saying something so fucking clever. No you're not.

Why my hate for "foodies" - they're just another niche of yuppie bourgeosie scum. They're snobby, they focus on the brands and labels. Oh you made porchetta, was it with Mangalitsa or Kurobuta pork? Oh, you made it with that factory stuff, oh well, no thanks then. Fuck you fucking status-conscious bitches. They're frequently condescending while often being incredibly wrong because they learn from fucktards like Food Network; they'll see things like "oh you bought fat asparagus, I only buy the thin stuff" well fuck you fucking condescending dicktard the fat stuff is the good stuff when it's the first spring growth. They're gossips and trend followers; they read the blogs and are always cooking things in the new "right way" to do things. They don't have real love for the ingredients and the history and the flavors and the technique. They spend a fortune on fancy cookware and knives but don't know how to julienne or rock the blade.

In the end the reason why I hate foodies so much is that they're so close to something that I love so dearly, and yet they spoil everything good about it. And they sometimes make food that's better than mine and that really does it.

05-25-09 | Using DOT graphviz

I wrote a while ago about DOT, the graphviz svg thing.

I thought I'd write a quick advice on using it from what I've learned. The manual dotguide.pdf and the online help are okay, so start there, but keep this in mind :

DOT was designed to make graphs for printing on paper. That leads to some weird quirks. For one thing, all the sizes are in *inches*. It also means that all the fitting support and such is not really what you want, so just disable all of that.

I have the best success when I use the Google method (what I stole from pprof in perftools). Basically they just set the font size and then let DOT make all the decisions about layout and sizing. There's one caveat in that : the way that DOT writes its font size is not supported right by FF3. See : 1 or 2 . There's a pretty easy way to fix this, basically for every font size tag, you need to add a "px" on it. So change "font-size:14.00;" to "font-size:14.00px;" . This change needs to be done on the SVG after dot. I do it by running a grep to replace ",fontcolor=blue" with "px". So in the DOT code I make all my text "blue", it's not actually blue, the grep just changes that to the px I need for my font sizes. So you'll see me output text attributes like "fontsize=24,fontcolor=blue".

The other big thing is that DOT seems to have zero edge label layout code. And in fact the edge layout code is a bit weak in general. It's pretty good at sizing the nodes and positioning the nodes - it has lots of fancy code for that, but then the edge labels are just slapped on, and the text will often be in horrible places. The solution to this is just to use edge labels as little as possible and make them short.

Another trick to getting better edge positioning is to explicitly supply the edge node ports and put them on a record. I do this in my huffman graphs. The other good edge trick is to use edgeweight. It doesn't change the appears of the edge, it changes how the edge is weighted for importance in the layout algorithm, so it becomes straighter and shorter.

For educational purposes I just made a little program to parse MSVC /showIncludes output to DOT :

my dotincludes zip .
dotincludes sample output from running on itself.

Here are some Huffman trees (SVG - you need a FireFox 3+ to view nicely) optimized with various Lagrange multipliers :

Huff 1
Huff 2
Huff 3

Remember it's control-wheel to zoom SVG's in FF.

05-22-09 | A little more Huffman

.. okay since I got started on this, a few other notes. This is ancient stuff, in fact it's been in the "Huffman2" in crblib for 10+ years, and it used to be common knowledge, but it's some of that art that's sort of drifted away.

For one thing, Huffman codes are specified only by the code lens. You don't need to store the bit patterns. Of course then you need a convention for how the bit patterns are made.

The "canonical" (aka standard way) to make the codes is to put numerically increasing codes in order with numerically increasing symbols of the same code length. I helps to think about the bit codes as if the top bit is transmitted first , even if you don't actually do that. That is, if the Huffman code is 011 that means first we send a 0 bit, then two ones, and that has value "3" in the integer register.

You can create the canonical code from the lengths in O(N) for N symbols using a simple loop like this :

for( len between min and max )
    nextCode = (nextCode + numCodesOfLen[len]) << 1;
    codePrefix[len] = nextCode;

for( all symbols )
    code[sym] = codePrefix[ codeLen[sym] ];
    codePrefix[ codeLen[sym] ] ++;

this looks a bit complex, but it's easy to see what's going on. If you imagine that you had your symbols sorted first by code length, and then by symbol Id, then you are simply handing out codes in numerical order, and shifting left one each time the code length bumps up. That is :

curCode = 0;
for( all symbols sorted by codelen : )
    code[sym] = curCode;
    curCode ++;
    curCode <<= codeLen[sym+1] - codeLen[sym];

An example :

code lens :

c : 1
b : 2
a : 3
d : 3


c : [0]
    code ++ [1]
    len diff = (2-1) so code <<= 1 [10]
b : [10]
    code ++ [11]
    len diff = (3-2) so code <<= 1 [110]
a : [110]
    code ++ [111]
    no len diff
d : [111]

very neat.

The other thing is to decode you don't actually need to build any tree structure. Huffman trees are specified entirely by the # of codes of each length, so you only need that information, not all the branches. It's like a balanced heap kind of a thing; you don't actually want to build a tree structure, you just store it in an array and you know the tree structure.

You can do an O(H) (H is the entropy, which is the average # of bits needed to decode a symbol) decoder using only 2 bytes per symbol (for 12 bit symbols and max code lengths of 16). (note that H <= log(N) - a perfectly balanced alphabet is the slowest case and it takes log(N) bits).

a node is :

  CodeLen : 4 bits
  Symbol  : 12 bits

You just store them sorted by codelen and then by code within that len (aka "canonical Huffman order"). So they are sitting in memory in same order you would draw the tree :

0 : c
10 : b
11 : a

would be

[ 1 : c ]
[ 2 : b ]
[ 2 : a ]

then the tree descent is completely determined, you don't need to store any child pointers, the bits that you read are the index into the tree. You can find the node you should be at any time by doing :

index = StartIndex[ codeLen ] + code;

StartIndex[codeLen] contains the index of the first code of that length *minus* the value of the first code bits at that len. You could compute StartIndex in O(1) , but in practice you should put them in a table; you only need 16 of them, one for each code len.

So the full decoder looks like :

code = 0;
codeLen = 0;

    code <<= 1;
    code |= readbit();
    codeLen ++;

    if ( Table[ StartIndex[ codeLen ] + code ].codeLen == codeLen )
        return Table[ StartIndex[ codeLen ] + code ].Symbol

obviously you can improve this in various ways, but it's interesting to understand the Huffman tree structure this way.

The thing is if you just list out the huffman codes in order, they numerically increase :


= 0, 2, 6,7

If you have a prefix that doesn't yet decode to anything, eg if you have "11" read so far - that will always be a code value that numerically doesn't exist in the table.

Finally, if your symbols actually naturally occur in sorted order, eg. 0 is most likely, then 1, etc.. (as they do with AC coefficients in the DCT which have geometric distribution) - then you can do a Huffman decoder that's O(H) with just the # of codes of each length !

05-22-09 | Fast Means

I know of three fast ways to track the mean of a variable. At time t you observe a value x_t. You want to track the mean of x over time quickly (eg. without a divide!). I generally want some kind of local mean, not the full mean since time 0.

1. "weighted decay". You want to do something like M_t = 0.9 * M_t-1 + 0.1 * x_t;

int accum = 0;

accum := ((15 * accum)>>4) + x_t;

mean = (accum>>4);

This is the same as doing the true (total/count) arithmetic mean, except that once count gets to some value (here 16) then instead of doing total += x_t, instead you do total = (15/16) * total + x_t; so that count sticks at 16.

2. "sliding window". Basically just track a window of the last N values, then you can add them to get the local mean. If N is power of 2 you don't divide. Of course the fast way is to only track inputs & exits, don't do the sum all the time.

int window[32];
int window_i = 0;
int accum = 0;

accum += x_t - window[window_i];
window[window_i] = x_t;
window_i = (window_i + 1) & 0x1F;

mean = (accum>>5);

3. "deferred summation". This is the technique I invented for arithmetic coding long ago (I'm sure it's an old technique in other fields). Basically you just do a normal arithmetic mean, but you wait until count is a power of two before doing the divide.

int accum = 0;
int next_count = 0;
int next_shift = 3;

accum += x_t;

if ( --next_count == 0 )
    mean = accum >> next_shift;
    accum = 0;
    next_shift = MIN( 10 , next_shift+1 );
    next_count = 1 << next_shift;

mean is not changing during each chunk, basically you use the mean from the last chunk and wait for another power-of-2 group to be done before you update mean. (also this codelet is resetting accum to zero each time but in practice you probably want to carry over some of the last accum).

For most purposes "deferred summation" is actually really bad. For the thing I invented it for - order 0 coding of stable sources - it's good, but that application is rare and not useful. For real context coding you need fast local adaptation and good handling of sparse contexts with very few statistics, so delaying until the next pow-2 of symbols seen is not okay.

The "weighted decay" method has a similar sort of flaw. The problem is you only have one tune parameter - the decay multiplier (which here became the shift down amount). If you tune it one way, it adapts way too fast to short term fluctuations, but if you tune it the other way it clings on to values from way way in the past much too long.

The best almost always is the sliding window, but the sliding window is not exactly cheap. It takes a lot of space, and it's not awesomely fast to update.

05-21-09 | Entropy Coding by Counting

There have been a number of interesting works in the last few years about different ways to do coding. Some refereneces :

L. Oktem : Hierarchical Enumerative Coding and Its Applications in Image Compressing, Ph.D. thesis
"Group Testing for Wavelet-Based Image Compression"
"Entropy Coding using Equiprobable Partitioning"
"Binary Combinatorial Coding"

The common pattern in all these is the idea of coding probabilistic events using counting. Basically, rather than taking some skew-probability event and trying to create a code whole length matches the probabilities like L = - log2(P) , instead you try to combine events such that you create coding decision points that have 50/50 or at least power of 2 probabilities.

Let's look at a simple example. Say you want to code a bunch of binary independent events, some 001100 sequence with a certain P(1) and P(0) that's constant. For now let's say p = P(1) is very low. (this is common case in image coding, or really any predictive coding where the prediction is very good - a 1 means the prediction was wrong, 0 means it was right; eg. high bands of wavelets are like this (though they aren't independent - more on that later)).

If you just try to code each bit one by one, the average length should be much less than one and you'd have to arithmetic code or something. To create an event that's just 50/50 we code a group of symbols. We want to make the probability that the whole group is off be close to 50%, so :

0.50 = P(0) ^ N

N = log(0.50) / log(P(0)) = -1 / log2( P(0) )

for example if P(1) = 5% , then N = 13.5

so you get 13 or 14 symbols then code are they all on or off with just 1 raw bit. That's an efficient coding operation because by design you've made that a 50/50 event. Now we have a bit of a puzzle for the next step. If you sent "it's all off" then you're done, but if you sent "something is on" you now have to send more data to specify what is on - but you have already sent some information by sending that something is on so you don't want to just waste that.

There's some crazy difficult combinatorics math to do if you wanted to figure out what the next event to code would be that was a perfect 50/50, but in the end our hand is forced because there's not really anything else we can do - we have to send a bit to say "is the first half on?". I'll present the group testing method without justification then try to justify it after the fact :

You know something in the group of N is on. First send a bit for whether the first half ([N/2]) range is on. If that's on, recurse onto that range and repeat. If the first half was on, you still have to move to your neighboring (N/2) which you don't know if it's on or off. If the first half was off, then move to the the neighbor, he must be on, so recurse into him without sending a bit.

Now the justification. First see if there was only 1 value on in the range N, then it would have equal probability of being in the first or second half of the group, so it would be correct to send a bit to specify if it was in the first or second half - that would be equivalent to singalling if the first half is active. Now note that by design of our group size it's by far most likely that there is only 1 value on. For our example case above where P(1) = 5%, the chance of there being exactly 1 value on is around 70%. In that case our code is perfect; for higher values on it's not ideal, but those are less likely, so the net loss is small.

(BTW the binary code to locate a single item in a group is just the binary coding of the index; eg. first you send a bit to indicate if it's in the top half or bottom half, etc.)

The enumerative coding approaches take a similar but slightly different tack. Basically they code the # of values on in the sequence. Once you have the # of values on, then all arrangements are equally likely, so you just need to code the permutation id with a fixed length equiprobable code.

Say for example you want to code 16 binary events

You first send the # on.

If there's 1 on, there are 16 ways for that to happen - just send 4 bits

If there're 2 on, there are 16*15/2 = 120 ways
    you should send a 6.906 bit code, but you can just send a 7 bit code

If there're 3 on, you want to send one of 16*15*14/6 = 560 ways
    a 10 bit code would suck here, so you want a simple prefix code
    that usually writes 9 bits and occasionally 10

This is pretty efficient, particularly when the probability of a bit being on is very low so that you are usually in the low-count cases. In the high count cases it starts to suck because counting the permutations gets pretty complex. The other problem with this method is how to encode the # that are on. The problem is the probability of some # being on is a binomial distribution. There's no simple fixed-bit code for a binomial distribution (I don't think).

Now, for N large, the binomial distribution approaches a normal or geometric distribution. In that case we can use a Golomb code for the # of values on. Furthermore for N large, the coding inefficiency of the # on becomes less important. In the real world I don't think N large is very practical.

Anyhoo I'm not sure any of this is super useful because in practice bits are never independent, so you want to do context coding to capture that, and it's a mess to do with these methods. The GTW (Group Testing for Wavelets) guys capture context information by classifying each sample into classes of similar expected statistics, and also by interleaved scanning so that coefficients with relations are not put into the same class together. The result is impressive, they get compression comparable to ECECOW, but all the work of shuffling values around and multiple scans means they're surely slower than just arithmetic coding.

BTW not related but while I was looking at this I was confused by the following stupidity :

Exponential Distribution == Laplacian == Bernoulli Process == Geometric

All of them are P(n) ~= r ^ n

(or P(n) ~= e ^ ( - lambda * n ) if you prefer)

~ means proportional, ignoring normalization.

(Poisson is similar but different - P(n) ~= r ^ n / n! )

BTW the maximum likelihood estimate of the laplacian parameter is the L1 norm

P(n) ~= e ^ ( - lambda * n )  ; lambda = 1 / (L1 norm of samples)

yeah yeah some of them are continuous and some discrete, some are one sided, some two sided, but people in compression play fast & loose with all of those things, so it's very confusing when you switch from one paper to another and they switch terms. Most people in compression describe the AC coefficients as having a "Laplacian" distribution.

BTW Golomb codes are optimal for coding Laplacian/Geometric distributions.

Mean of geometric distribution with ratio r is = r / (1.0 - r)
  (r given the mean is r = m / (1+m) )

The optimal Golomb parameter k satisfies the relation :

r^k + r^(k+1) <= 1 < r^k + r^(k-1)


r^k <= 1/(1+r) < r^(k-1)

You can find this k a few ways :

k = froundint( 0.5 - log(1+r)/log(r) );

k = ceil( log(1+r)/log(1/r) );


int GetK(double r)
    double pk = p;
    for(int k =1;;k++)
        if ( (1.0 + p)*pk <= 1.0 )
            return k;
        pk *= p;

and furthermore we often use the nice approximation

k = froundint( mean )

however, that is not really correct for large mean.  It works fine for mean up to around 5.

At mean = 6 , the true optimum k is 5

true entropy : 4.144196
k=5 , Golomb H : 4.174255
k=6 , Golomb H : 4.219482

you can see at k=5 Golomb does very well matching entropy, but using the incorrect k= mean it gets off.

At larger mean it's much worse -

at mean = 19, the correct k is = 14

true entropy : 5.727806
golomb_k : 14 , H : 5.761475
golomb_k : 19 , H : 5.824371

At low mean of course Golomb gets bad because it can't code the 0 in less than 1 bit. You would obviously use run-lenghts or something to fix that in practice. Golomb works very well down to a mean of about 1.0 (that's a geometric r of 0.5).

One neat thing about geometric distributions is if you code the first symbol a different way, the remainder is still geometric. So you can code the 0 symbol with RLE, then if it's not 0, you code the symbols >= 1 with Golomb, and it's still geometric with the same r and k. (though Golomb is still bad then it's being done less often).

BTW I tried expressing k in terms of the mean :

k = ceil( log(1+r)/log(1/r) );

k = ( log( (1+2m) / (1+m))/log((1+m)/m) )

k = ( log( 1+2m ) - log(1+m) ) / ( log (1+m) - log(m) )

which is sort of handy if you have 'm' as an integer you can use table lookups for the logs and just do one divide, but it's
not really super simple.

what I'd like is just a Taylor expansion of this that works for m < 30 or so.  In particular it should be something like

k = m - 0.2 * m^2  (but not this)

k is like m for m small, but then it needs to correct down.  I couldn't work it out, maybe you can.

05-20-09 | Some Huffman notes

For the record, you can do a very fast Huffman decoder by always reading 8 bits and storing the bit-reading state in the table pointer. The way you do this is by creating a seperate 8-bit lookup table for each possible current bit-reading state.

You have some # of bits [0,8] that have not yet been used for output. For each # of bits there are various values those bits can have, for 0 bits its {} , for 1 bit it's {0,1}, etc. So your total number of states is 256+128+64... = 512.

So at each step you have your current table pointer which is your decode table plus holds your current bit buffer state. You read 8 more bits and look up in that table. The table tells you what symbols can be made from the combination of {past bit buffer + new 8 bits}. Those symbols may not use up all the bits that you gave it. Whatever bits are left specify a new state.

I'll show an example for clarity :

Say you read 3 bits at a time instead of 8 (obviously we picked 8 because it's byte aligned)

If your Huffman code is :

0 - a
10 - b
11 - c

You make these tables :

{no bits} :

000 : aaa + {no bits}
001 : aa  + {1}
010 : ab  + {no bits}
011 : ac  + {no bits}
100 : ba  + {no bits}
101 : b   + {1}
110 : ca  + {no bits}
111 : c   + {1}

{1} :  (means ther's 1 bit pending of value 1)

000 : baa + {no bits}
001 : ba  + {1}
010 : bb  + {no bits}
011 : bc  + {no bits}
100 : caa + {no bits}
101 : ca  + {1}
110 : cb  + {no bits}
111 : cc  + {no bits}

The decoder code looks like this :

struct DecodeTableItem
    U8 numOut;
    U8 output[MAX_OUT]; 
    DecodeTableItem * nextTable;

DecodeTableItem * pTable = table_no_bits_pending;

    U8 byte = *inptr++;

    int num = pTable[byte].numOut;

    for(int i=0;i < num;i++)
        *outptr++ = pTable[byte].output[i];
    pTable = pTable[byte].nextTable;

It's actually a tiny bit more complicated than this because you have to handle codes longer than 8 bits. That just means "numOut" is 0 and you have to fall out to a brute force loop. You could handle that case in this same system if you make tables for 9,10,11+ bits in the queue, but you don't want to make all those tables. (you also don't actually need to make the 8 bits in the queue table either). The "numOut = 0" loop will read more bytes until it has enough bits to decode a symbol, output that symbol, and then there will be 0-7 remainder bits that will select the next table.

BTW in practice this is pretty useless because we almost never want to do one giant huffman array decode, we have our huffman codes interleaved with other kinds of coding, or other huffman trees, dependent lookups, etc. (this is also useless because it's too many tables and they are too slow to make on the fly unless you are decoding a ton of data).

BTW old-school compression people might recognize that this is basically a Howard-Vitter Quasi-Arithmetic coder. In the end, every decompressor is just a state machine that's fed by input bits or bytes from the coded stream. What we are doing here is basically explicitly turning our algorithm into that state machine. The Howard-Vitter QA did the same thing for a simplified version of arithmetic coding.

There are a lot of wins for explicitly modeling your decoder as a state machine triggered on input bits. By design of the compressor, the bits in the coded stream are random, which means any branch on them in the decoder is totally unpredictable. You have to eat that unpredictable branch - but you only want one of them, not a whole bunch. In normal function-abstracted decompressors you read the bits in some coder and generate a result and then act on that result outside - you're essentially eating the branch twice or more.

A typical arithmetic decoder does something like :

Read bits from stream into "Code"

Test "Code" against Range and probabilities to find Symbol
    -> this is the main branch on the bit stream

if ( Range too small )
    do Renormalization

branch on Symbol to update probabilities

do conditional output using Symbol
    eg. Symbol might be a runlength or something that makes us branch again

The state-based binary arithmetic coders like the QM or MQ coder combine the bit-reading, renormalization, and model updates into a single state machine.

05-18-09 | Lagrange Space/Speed Optimization

We just did an interesting thing at RAD that's kind of related to what I've been writing about.

A while ago Dmitry Shkarin (ppmii inventor) posted this code sketch for his Huffman decoder :

Dmitry Shkarin's Huffman reader :

Suppose We have calculated lengths of rrHuffman codes and minimal
codelength is N then We can read N bits and to stop for most probable
symbols and to repeat reading for other symbols. Decoding procedure will
be something similar to:

extern UINT GetBits(UINT NBits);

INT_TYPE NextTable, BitsToRead;

inline UINT DecodeSymbol()
const HUFF_UNPACK_ITEM* ptr=Table;
do {
ptr += ptr->NextTable+GetBits(ptr->BitsToRead);
} while (ptr->BitsToRead != 0);
return ptr->NextTable;

this will seem rather opaque to you if you don't know about Huffman codes; you can ignore it and read on and still get the point.

Dmitry's decoder reads the minimum # of bits to get to the next resolved branch at each step, then increments into the table by that branch. Obviously the value of the bits you read is equal to the branch number. So like if you read two bits, 00 = 0, 01 = 1, 10 = 2, 11 = 3 - you just add the bits you read to the base index and that's the branch you take.

Okay, that's pretty simple and nice, but it's not super fast. It's well known that a good way to accelerate Huffman decoding is just to have a big table of how to decode a large fixed-size bit read. Instead of reading variable amounts, you always just read 12 bits to start (for example), and use that 12 bit value to look up in a 4096 member table. That table tells you how many bits were actually needed and what symbol you decoded. If more than 12 bits are needed, it gives you a pointer to a followup table to resolve the symbol exactly. The crucial thing about that is that long symbols are very unlikely (the probability of each symbol is like 2^-L for a bit length of L) so you rarely need the long decode path.

It's pretty obvious that you could extend Dmitry's method to encompass read-aheads like this acceleration table. Instead of just doing GetBits on "BitsToRead" , instead you scan ahead BitsToRead , and then when you take a path you add an extra field like "BitsConsumed" which tells you how many of those bits were actually needed. This lets you make initial jump tables that read a whole bunch of bits in one go.

More generally, in the tree building, at any point you could decide to make a big fast node that wastes memory, or a small binary treeish kind of node. This is kind of like a Judy tree design, or a Patricia Trie where the nodes can switch between linked-lists of children or an array of child pointers. One nice thing here is our decoder doesn't need to switch on the node type, it always uses the same decode code, but the tree is just bigger or smaller.

To be concrete here's a simply Huffman code and possible trees for it :

Huffman codes :

0   : a
10  : b
110 : c
111 : d

Possible trees :

(* means read one bit and branch)

Tree 1)

*----- a
  *--- b
    *- c

4 leaves
3 internal nodes
= 7 nodes total

Tree 2)

[ 00 ]--- a
[ 01 ]--- a
[ 10 ]--- b
[ 11 ]- *- c
         \ d

5 leaves
2 internal nodes
= 7 nodes total

Tree 3)

[3bit] -
[ 000 ] -- a
[ 110 ] -- c
[ 111 ] -- d

8 leaves
1 internal node
= 9 nodes total

Tree 4)

*------- a
  [ 00 ] - b
  [ 01 ] - b
  [ 10 ] - c
  [ 11 ] - d

5 leaves
2 internal nodes
= 7 nodes total

We have these four trees. They have different memory sizes. We can also make an estimate of what the decode time for each tree would be. In particular for the case of Huffman decoding, the expected time is something like the number of branches weighted by 2^-depth of each branch. Reading more bits in a given branch isn't significantly slower than reading 1 bit, we just want as few branches as possible to decode a symbol.

I'm going to get away from the Huffman details now. In general when we are trying to make fast data structures, what we want is as much speedup as possible for a given memory use. Obviously we could throw 64k of table slots at it and read 16 bits all at once and be very fast. Or we could use the minimum-memory table. Usually we want to be somewhere in between, we want a sweet spot where we give it some amount of memory and get a good speedup. It's a trade off problem.

If we tried all possible trees, you could just measure the Mem use & Time for each tree and pick the one you like best. You would see there is some graph Time(Mem) - Time as a function of Mem. For minimum Mem, Time is high, as you give it more Mem, Time should go down. Obviously that would be very slow and we don't want to do that.

One way to think about it is like this : start with the tree that consumes minimum memory. Now we want to let it have a little bit more memory. We want the best bang-for-the-buck payoff for that added memory, so we want the tree change that gives us the best speedup per extra byte consumed. That's the optimum d(Time)/d(Mem). Keep doing improvements until d(Time)/d(Mem) doesn't give you a big enough win to be worth it.

Some of you may already be recognizing this - this is just a Rate-Distortion problem, and we can solve it efficiently with a Lagrange multiplier.

Create a Lagrange cost like :

J = Time + lambda * Mem

Now try to build the tree that minimizes J for a given lambda. (ignore how lambda was picked for now, just give it some value).

If you find the tree that minimizes J, then that is the tree that is on the optimal Mem-Time curve at a certain spot of the slope d(Time)/d(Mem).

You should be able to see this is true. First of all - when J is minimum, you must be on the optimal Time(Mem) curve. If you weren't, then you could hold Mem constant and improve Time by moving towards the optimal curve and thus get a lower J cost.

Now, where are you on the curve? You must be at the spot where lambda = - d(Time)/d(Mem). One way to see this is algebraically :

J is at a minimum, therefore :

d(J)/d(Mem) = 0

d(J)/d(Mem) = d(Time)/d(Mem) + lambda * 1 = 0

therefore lambda = - d(Time)/d(Mem)

You can also see this intuitively. If I start out anywhere on the optimal Time(Mem) curve, I can improve J by trading Mem for Time as long as the gain I get in Time is exceeding lamda * what I lose in Mem. That is, if d(Time) > - lambda * d(Mem) , then I should do the step. Obviously you keep doing that until they are equal. QED.

Since I'm nice I drew a purty picture :

The blue curve is the set of optimal solutions for various Mem parameters. The hard to see yellow tangent is a lambda parameter which is selecting a specific spot on the trade-off curve. The green region above the curve is the space of all possible solutions - they are inefficient solutions because you can improve them by getting to the blue curve. The red region is not possible.

This kind of lagrange space-time optimization has all sorts of applications in games. One example would be your spatial acceleration structures, or something like kD trees or BVH hierarchies for ray tracing. Too often we use hacky heuristics to build these. What you should really do is create a lagrange cost that weighs the cost of more memory use vs. the expected speedup.

One of the nice wins of this approach is that you can often get away with doing a greedy forward optimization for J (the lagrange cost), and it's just a single decision as you build your tree. You just evaluate your current choices for J and pick the best one. You do then have to retry and dial lambda to search for a given Mem target. If you didn't use the lagrange multiplier approach, you would have to try all the approaches and record the Time/Mem of every single possibility, then you would pick the one that has the best Time for the desired Mem.

In the past I've talked about algorithms being dumb because they are "off the curve". That means they are in the green region of the picture - there's just no reason to use that tradeoff. In general in algorithms you can't fault someone for selecting something on the blue curve. eg. a linked list hash table vs. a reprobing hash table might be at different spots on the blue curve - but they're both optimal. The only time you're being really retarded is when you're way out in green space of wanton inefficiency.

Back to the specifics of the Huffman decoder - what we've found is kind of interesting and maybe I'll write about it in more detail later when we understand it better. (or it might be one of our magic secrets).

ADDENDUM : I should be clear that the with the J-cost optimization you still have to consider greedy tree building vs. optimal tree building. What you're gaining is a way to drive yourself towards the blue curve, but it's not like the tree building is suddenly super easy. You also have to deal with searching around in the lagrange multiplier to hit the target you want. For many practical problems you can create experimental heuristics that will give you a formula for the lambda that gives you a certain mem use or rate or whatever.

When we make advances in the art, there are two main ways we do it. One is to push farther along the blue curve than anyone has before. For example in data compression we usually have a run time vs. compression curve. You can run slower and slower algorithms and get more and more compression. You might find a new algorithm that runs even slower and gets even more compression than anyone has before. (PAQ is playing this game now; I used to play this game with PPMZ back in the day). That extends the curve out into areas unexplored. The other advance is to find something that's below the previously known blue curve. You find a way to get a better trade-off and you set a new optimal curve point. You might find a new compressor that gets the same compression ratio as old stuff, but runs way faster.

05-15-09 | Trellis Quantization

There's a big gap in all technical literature. You can pretty easily find hand-wavey overviews of things for non-specialists in any field that make your feel all happy, but don't get into any details so they're useless if you actually need to do something in that field. Then there's also an abundance of technical papers that are steeped in the terminology and notation of that particular sub-field and are completely opaque. It's almost impossible to find transition material.

One case of this for me was so-called "Trellis Quantization". Part of the problem was that there's a whole body of old literature on "trellis coded quantization" which is an entirely different thing. TCQ is about designing non-linear or VQ quantizers for sources. Almost all of that research from the old days is totally worthless, because it assumed *NO ENTROPY CODING* on the post-quantization coefficients. Thus it was trying to make quantization buckets which had equal probability. Even at the time that research was happening it was already retarded, but it's especially retarded now. We know that in the presence of an entropy coder, independent variables are R-D optimal with a fix-step-size deadzone quantizer.

But in the real world our entropy coder is not independent. That is, we use things like run length codes, or adaptive arithmetic coders, or other schemes, such that the way we code one variable affects the code length for the next variable. This is where so-called "trellis quantization" in the modern sense comes in.

I really hate the modern use of the term "trellis quantization" because it's really not about the trellis or the quantization. A better term would be "dynamic programming code stream output optimization". If somebody had just told me that's what it was at the beginning it would have saved me weeks of confusion. It's basically the same thing you do in an LZ optimal parser (though not exactly).

This technique is used mainly in lossy image coders to optimize the coding of a certain bunch of pixels. It can be used for bitplane truncation in zerotree type coders, but it's mainly used for block-transforms, and mainly for video coders (it was made prominent in H263 and is used in H264).

Basically it goes like this : for a given block, you do normal quantization and you get a bunch of coefficients like {17,3,9,4,0,1,2,0}. You could just output that, and you would have a certain rate and distortion. But you could also change some of those coefficients. That would dicrease your rate and increase your distortion. (note that the increase in distortion may be very small in some cases - if the original values were very close to a quantization bucket edge, then you can shift them without much pain). You might be able to output a better Rate-Distortion optimal block by choosing some other output, such as {17,3,9,4,0,1,1,0} or {17,0,8,4,0,0,0,0} depending on where you are on the R-D curve.

The "trellis" comes in because you don't need to consider all possible outputs. It's easiest to think of like this :

Say you have 4 binary coding decisions. That is, you could make choices 0000 or 0001, etc. there are 16 possible choices. Naively, you would have to consider all 16 sequences and rate each one and pick the best. If you draw this as a graph, it looks like a tree - each node has two kids, it branches out and you have 16 leaves. But in many coding scenarios, your current coding cost does not depend on the entire history of your sequence - it only depends on the current state. For example, say you are doing order-1 context modeling and binary arithmetic encoding. Then there is a cost to encode a 0 after a 0, a 0 after a 1, a 1 after 0 and a 1 after a 1 (c00,c01,c10,c11). Each path in the graph is a coding action. The graph you need to consider is like this :

  [ 1 ]----[ 1 ]----[ 1 ]----[ 1 ]
 /     \  /     \  /     \  /     \ 
/       \/       \/       \/       \
\       /\       /\       /\       / 
 \     /  \     /  \     /  \     /  
  [ 0 ]----[ 0 ]----[ 0 ]----[ 0 ]

you start at the far left, as you go along each edge that's a coding output. Any given state, it doesn't matter how you got there, the transitions out of it have the same cost regardless of history. To fill out the graph you start on the far left with a cost of zero, you walk each link, and you fill in the node if the cost you are carrying is lower than what's already in there. Each node only needs to remember the cheapest way to get to that node.

To find the optimal coding you start at the far right and you walk backwards along the links that led you to the cheapest cost.

This graph looks like a "trellis" so these kind of problems are called "trellis quantization" or "joint trellis subband optimization" or whatever. The key thing about the trellis shape is that for N coding decisions the size of the graph is O(K*N) (if there are K options at each decision point), whereas the full branching factor is K^N if you had to consider all possible sequences.

This is apparently the "Viterbi algorithm" or some such thing but all the descriptions I've found of that are weird and confusing.

For game developers, this is very similar to A* path finding. A* is actually a form of Lazy Dynamic Programming. Path finding has the character we need because the cost to go between two nodes only depends on those two nodes, not the whole history, thus at each node you only need to remember the shortest path to get to that node.

In H264 this is a nice way to really optimize the output for a given block. It's sort of weirdly called "quantization" because it often consists of jamming values to zero which is kind of like using a larger adaptive deadzone. I really don't like that terminology, because it is in fact NOT a variable quantizer, since it doesn't affect the reconstruction levels in the decoder at all.

Note that in video coding this is a very small bit of a tiny optimization, and the rest of the optimization is a big heuristic mess. The total optimization looks something like this :

Assign bit rate to various frames
  (try to optimize R-D ; meet channel overflow and buffer underflow constraints )

Within a frame, assign bits to motion vectors vs. residuals & control codes

Iterate :

    choose best motion vectors for current motion vector bit rate allocation

    optimize block mode decisions

    code residuals to given bit allocation

        "trellis" optimize the coding of each block to a lagrangian R-D (R + lambda * D)

    oh and also iteratively search around on the lagrange multiplier lambda
        to hit the rate you were supposed to

    then hold residuals constant and optimize block & motion vector decisions for those residuals

  then shift bit allocation between modes, residuals & vectors & repeat

yuck. Oh, and of course per-block coding can't really be independently optimized since context model state carries between blocks. And in intra frames blocks are used to predict other blocks, so if you change one it affects all future ones. Oh, and this optimization is super important.

05-14-09 | Image Compression Rambling Part 2

Okay we talked a bit about block transforms, now let's talk about some of the somewhat weird variants of block transforms that are used in modern standard coders.

With an 8x8 block we're at a big disadvantage. An 8x8 block is like a 3 level wavelet. That's not much, wavelet coders rely on a 5 or 6 level transform normally, which would correspond to a 32x32 block or better. Large block transforms like that are bad because they're computationally complex, but also because they are visually bad. Large blocks create worse blocking artifacts, and also increase ringing, because it makes the high frequency shapes very non-local.

Basically by only doing 8x8 we are leaving a lot of redundancy between neighboring blocks. There's moderate correlation within a block, but also strong correlation across blocks for coefficients of the same type.

H264 Intra frame coding is actually really excellent; it outperforms JPEG-2000 for example. There're a few papers on this idea of using H264 intra coding just for still images, and a project called AIC . (AIC performs worse than real H264 for a few reasons I'll get into later).

"AIC" basically just does 8x8 block DCT's - but it does this interesting thing of pre-predicting the block before the transform. It works on blocks in scan order, and for each block before it does the DCT it creates a prediction from the already transmitted neighbors and subtracts that off. This is a nice page with the details . What this accomplishes does is greatly reduce correlation between blocks. It subtracts off predicted DC so the DC is usually small, and also often subtracts off predicted shapes, so for example if you're in a smooth gradient region it subtracts off that gradient.

Real H264 intra beats "AIC" pretty well. I'm not sure exactly why that is, but I have a few guesses. H264 uses integer transforms, AIC uses floating point (mainly a big deal at very high bit rates). H264 uses macroblocks and various sub-block sizes; in particular it can choose 8x8 or 4x4 sub-blocks, AIC always uses 8x8. Choosing smaller blocks in high detail areas can be a win. I think the biggest difference is probably that the H264 implementations tested do some RDO while AIC does not. I'm not sure exactly how they do RDO on the Intra blocks because each block affects the next one, but I guess they could at least sequentially optimize each block as they go with a "trellis quantizer" (see next post on this).

Okie doke. JPEG XR has similar issues but solves them in different ways. JPEG XR fundamentally uses a 4x4 transform similar to a DCT. 4x4 is too small to remove a lot of correlation, so neighboring blocks are very highly correlated. To address this, JPEG XR groups 4x4 groups of blocks together, so it has a 16x16 macroblock. The DC's of each of the 4x4 blocks gets another pass of the 4x4 transform. This is a lot like doing a wavelet transform but getting 4:1 reduction instead of 2:1. Within the 16x16 macroblock, each coefficient is predicted from its neighbor using gradient predictors similar to H264's.

In H264 the gradient predictor is chosen in the encoder and transmitted. In JPEG XR the gradient predictor is adaptively chosen by some decision that's made in the encoder & decoder. (I haven't found the exact details on this). Also in JPEG XR the delta-from-prediction is done *post* transform, while in H264 it was done pre-transform.

If you think about it, there's a whole world of possibilities here. You could do 4x4 transforms again on all the coefficients. That would be very similar to doing a 16x16 DCT (though not exactly the same - you would have to apply some twiddle factors and butterflies to make it really the same). You could do various types of deltas in pre-transform space and post-transform space. Basically you can use the previous transmitted data in any way you want to reduce what you need to send.

One way to think about all this is that we're trying to make the reconstruction look better when we send all zeros. That is, at low bit rates, we will very often have the case that the entire block of AC coefficients goes to zero. What does our output look like in that case? With plain old JPEG we will make a big solid 8x8 block. With H264 we will make some kind of gradient as chosen by the neighbor predictor mode. With JPEG XR we will get some predicted AC's values untransformed, and it will also be smoothed into the neighbors by "lapping".

So, let's get into lapping. Lapping basically gives us a nicer output signal when all the AC's are zero. I wrote a bit about lapping before . That post described lapping in terms of being a double-size invertable transform. That is, it's a transform that takes 2N taps -> N coefficients and back -> 2N , such that if you overlap with neighbors you get exact reconstruction. The nice thing is you can make it a smooth window that goes to zero at the edges, so that you have no hard block edge boundaries.

Amusingly there are a lot of different ways to construct lapped transforms. There are a huge family of them (see papers on VLGBT or some fucking acronym or other). There are lots of approaches that all give you the same thing :

2N -> N windowed basis functions as above
  (nobody actually uses this approach but it's nice theoretically)

Pre & post filtering on the image values (time domain or spatial domain)
  basically the post-filter is a blur and the pre-filter is a sharpen that inverts the blur
  (this can be formulated with integer lifting)

Post-DCT filtering (aka FLT - Fast Lapped Transform)
  basically do the NxN DCT as usual
  then swizzle the DCT coefficients into the neighboring DCT's

Post-DCT filtering can either be done on all the coefficients, or just on the first few (DC and primary AC coefficients).

Lapping is good and bad. It's not entirely an awesome win. For one thing, the pre-filter that the lap does is basically a sharpen, so it actually makes your data harder to compress. That's sort of balanced by having a better reconstruction shape for any given bit rate, but not always. The fundamental reason for this is that lapping relies on larger local smoothness. eg. for 8x8 blocks you're doing 16-tap lapped transforms. If your signal is actually smooth over 16 taps then it's all good, but when it's not, the lapped transform needs *larger* AC coefficients to compensate than a plain blocked 8-tap DCT would.

The interesting thing to me is to open this up and consider our options. Think just about the decoder. When I get the DC coefficient at a given spot in the image - I don't need to plop down the shape of any certain transform coefficient. What I should do is use that coefficient to plop down my best guess of what the pixels here were in the original. When I get the next AC coefficient, I should use that to refine.

One way to think about this is that we could in fact create an optimized local basis. The encoder and decoder should make the same local basis based on past transmitted data only. For example, you could take all the previously sent nearby blocks in the image, run PCA on them to create the local KLT ! This is obviously computationally prohibitive, but it gives us an idea of what's possible and how far off. Basically what this is doing is making the DC coefficient multiply a shape which is our best guess for what the block will be. Then the 1st AC coefficient multiplies our best guess for how the block might vary from that first guess, etc.

05-15-09 | Image Compression Rambling Part 3 - HDR

Next part : HDR (High Dynamic Range) coding. One of the things I'd like to do well for the future is store HDR data well, since it's more and more important to games.

The basic issue is that the range of values you need to store is massive. This means 8-bit is no good, but really it means any linear encoding is no good. The reason is that the information content of values is relative to their local average (or something like that).

For example, imagine you have an image of the lighting in a room during the day with the windows open. The areas directly hit by the sun are blazing bright, 10^6 nits or whatever. The areas that are in shadow are very dark. However, if you rotate and look at the image with your back to the sun, your eyes adjust and you can see a lot of detail in the shadow area still. If you encoded linearly you would have thrown that all away.

There are a few possibilities for storing this. JPEG-XR (HD Photo) seems to mainly just advocate using floating point pixels, such as F16-RGB (48 bits per pixel). One nice thing about this "half" 48 bit format is that graphics cards now support it directly, so it can be used in textures with no conversion. They have another 32 bit mode RGBE which uses an 8-bit shared exponent and 3 8-bit mantissas. RGBE is not very awesome; it gives you more dynamic range than you need, and not enough precision.

I haven't been able to figure out exactly how they wind up dealing with the floating point in the coder though. You don't want to encode the exponent and mantissa seperately, because the mantissa jumps around weirdly if you aren't aware of the exponent. At some point you need to quantize and make integer output levels.

One option would be do a block DCT on floating point values, send the DC as a real floating point, and then send the AC as fixed-size steps, where the step size is set from the magnitude of the DC. I think this is actually an okay approach, except for the fact that the block artifacts around something like the sun would be horrific in absolute value (though not worse after tone mapping).

Some kind of log-magnitude seems natural because relative magnitude is what really matters. That is, in areas of the image near 0, you want very small steps, but right near the sun where the intensity is 10^8 or whatever you only need steps of 10^5.

This leads to an interesting old idea : relative quantization. This is most obvious on a DPCM type coder. For each pixel you are encoding, you predict the pixel from the previous, and subtract from the prediction and send the error. To get a lossy coder, you quantize the error before encoding it. Rather than quantize with a fixed size step, you quantize relative to the magnitude of the local neighborhood (or the prediction). You create a local scale S and quantize in steps of {S,2S} - but then you also always include some very large steps. So for example in a neighborhood that's near black you might use steps like {1,2,3,4,5...16,20,...256,512,1024,...10^5,10^6,10^7}. You allow a huge dynamic range step away from black, but you don't give many values to those huge steps. Once you get up into a very high magnitude, the low steps would be relative.

This takes advantage of two things : 1. we see variation relative to the local average, and 2. in areas of very large very high frequency variation, we have extremely little sensitivity to exactly what the magnitude of that step is. Basically if you're look at black and you suddenly look at the sun you can only tell "that's fucking bright" , not how bright. In fact you could be off by an order of magnitude or more.

Note that this is all very similar to gamma-correction and is sort of redundant with it. I don't want to get myself tangled up in doing gamma and de-gamma and color space corrections and so on. I just want to be a pixel bucket that people can jam whatever they want in. Variable step sizing like this has been around as an idea in image compression for a very long time. It's a good basic idea - even for 8 bit low dynamic range standard images it is interesting, for example if you have a bunch of pixels like {1,3,2,1,200,1,3,2} - the exact value of that very big one is not very important at all. The bunch of small ones you want to send precisely, but the big one you could tolerate 10 steps of error.

While that's true and seems very valuable, it's very hard to use in practice, which is why no one is doing it. The problem is that you would need to account for non-local effects to actually get away with it. For example - changing a 200 to a 190 in that example above might be okay. But what if that was a row if pixels, and the 200 is actually a vertical edge with lots of pixel values at 200. In that case, you would be able to tell if a 200 suddenly jumped to a 190. Similarly, if you have a bunch of values like - {1,3,2,1,200,200,200,1,3,2} - again the eye is very bad at seeing that the 200 is actually a 200 - you could replace all three of those with a 190 and it would be fine. However if your row was like this : {1,3,2,1,200,200,200,1,3,2,1,200,200,2} - and you changed some of the 200's but not the others - then that would be very visible. The eye can't see absolute intensity very well at all, but it can see relative intensity and repetitions of a value, and it can see edges and shapes. So you have to do some big nonlocal analysis to know how much error you can really use in any given spot.

Greg Ward has an interesting approach for backward compatible JPEG HDR . Basically he makes a tone-mapped version of the image, stores that in normal RGB, divides that by the full range original image, and stores the ratio in a seperate gray scale channel. This is a good compromise, but it's not how you would actually want to store HDR if you had a custom pixel format. (it's bad because it presumes some tone mapping, and the ratio image is very redundant with the luma of the RGB image).

I'm leaning towards Log-Y-UV of some kind. Probably with only Log(Y) and UV linear or something like that. Greg Ward has a summary of a bunch of different HDR formats . Apparently game devs are actually using his LogLuv : see Christer or Deano or Matt Pettineo & Marco .

One nasty thing about LogLuv is that there's some problems near zero, and everybody is doing slightly different hacks to deal with that. Yay. It also doesn't bilerp nicely.

ADDENDUM : I should clarify cuz this got really rambly. LogLuv is an *integer* encoding. As a pixel bucket I can of course handle 3 integers of various scales and signs, no problemo. Also, "floating points" that are in linear importance scale are not really a difficult issue either.

The tricky issue comes from "real" floating points. That is, real HDR image data that you don't want to integerize in some kind of LogLuv type function for whatever reason. In that case, the actual floating point representation is pretty good. Storing N bits of mantissa gives you N bits of information relative to the overall scale of the thing.

The problems I have with literally storing floating point exponent and mantissa are :

1. Redundancy beteen M & E = worse compression. Could be fixed by using one as the context for the other.

2. Extra coding ops. Sending M & E instead of just a value is twice as slow, twice as many arithmetic code ops, whatever.

3. The mantissa does these weird step things. If you just compress M on its own as a plane, it does these huge steps when the exponent changes. Like for the values 1.8,1.9,1.99,2.01,2.10 - M goes from 0.99 to 0.01. Then you do a wavelet or whatever and lossy transform it and it smooths out that zigzag all wrong. Very bad.

So clearly just sending M & E independently is terrible.

05-13-09 | Image Compression Rambling Part 1

What exactly is a block transform like a DCT or whatever, and why do we do it? This is sort of rambling socratic questioning for my self.

We have some 1d signal of length N (that's just N floats). We want to transform it and preserve L2 norm. (Why exactly do we want to preserve L2 norm? It's not strictly necessary but it means that quantizing in transformed space is the same as transforming in pre-transform space which is handy.)

Well, preserving L2 norm is just the same as pretending our signal is a vector in N-d space and preserving its length. That means that our transform is just a rotation (our transform is real and invertable). In particular an N-point transform is a member of SO(N).

That's most obvious in 2d so let's start there. I wrote before about the 2d Haar/Hadamard/S-transform and whatnot.

What are all the possible 2d matrices that can transform a signal and preserve length ?

I : (identity)
[1 0]
[0 1]

J : (exchange)
[0 1]
[1 0]

Z : (mirror)
[1  0]
[0 -1]

ZZ = I , JJ = I

K = -JZ = ZJ =
[0  1]
[-1 0]

KK = -1
K^T = -K
K^T K = 1

H = Hadamard = J + Z :
[1  1]
[1 -1]

is all you can make; (you can stick arbitrary factors of J or K on there, or -1's). Let's see what kind of linear combination we can make :

R(c,s) = c * I + s * K

R^T * R = ( c * I + s * K ) ^T * ( c * I + s * K ) 

R^T * R = ( c * I ^T + s * K ^T ) * ( c * I + s * K ) 

R^T * R = ( c * I - s * K ) * ( c * I + s * K ) 

R^T * R = ( c^2 * I - s^2 * K*K ) 

R^T * R = ( c^2 * I + s^2 * I ) 

R^T * R = ( c^2 + s^2 )  * I

therefore ( c^2 + s^2 ) = 1

therefore c & s are a cosine & sine and R is a rotation

You see a lot of signal processing papers drawing these fucking signal flow diagrams that I hate. One of the things they like to draw is a "butterfly" . There's some ambiguity about what people mean by a "butterfly". The Wikipedia page is using the convention that it's just a Hadamard. People will usually put a little "-" on the line that gets the negative to disambiguate.

Sometimes you'll see a variable written next to a line of a butterfly. That means multiply by that number. We saw in my earlier post how rotations can be made from shears. If you ignore normalization for a moment, we can see that 2d rotations can be made just by applying one of our matrices (such as H), then multiplying one variable by a scalar. In terms of circuit diagrams, I and J are just lines moving around, Z is a negative applied to one line, H is a butterfly. That's all you need to do any 2d rotation.

Some of the lapped transform people confusingly use the "mirrored rotation" matrix :

M(a) = Z * R(a)

M =
[ c   s ]
[ s  -c ]

M^T = M
M * M = I

M is its own transpose and its own inverse
btw you can see this geometrically because Z * R(a) = R(-a) * Z

Haar = M( 45 degrees ) = sqrt(1/2) * Hadamard

Now any N-d rotation can be made from a series of 2d planar rotations. That is in fact how you build the FFT.

Since any transform is a rotation, the 8-tap DCT must be an 8-d rotation. We could figure out all the angles by thinking about what it does to various vectors. All constant vectors [c,c,c,c,c,c,c,c] are rotated to [1,0,0,0,0,0,0,0] ; that's one angle, etc. Each of those rotations is just a 2d rotation in some plane or other.

We can bring this back to the PCA/KLT I wrote about before . The PCA finds the axes of principal variation. The KLT is then the rotation matrix which rotates the principal axes to the coordinate axes (that is, it's the spatial frame where the covariance is diagonal).

Now we can ask why one set of axes or the other. Apparently the DCT is the KLT for a simple model of neighbor-correlated data. (specifically, if the correlation matrix is symmetric and tri-diagonal). I'd like to find a simple proof of this, I haven't been able to find it yet. (help). Recently there have been some papers on using the Tchebichef transform (DTT) instead of the DCT. Personally I think this is rather moot because nobody just does transforms and sends the coefficients directly any more. We always take deltas from neighbors or use correlations in other ways, so having transforms that decorrelate more is somewhat irrelevant.

(BTW there is one big win with the DTT - it's based on polynomials so the first AC coefficient is in fact a straight line ramp; with the DCT the first AC is a cosine shape. For synthetic images this could be a big win because the DTT can capture linear gradients exactly in only 2 coefficients).

But let's back up a second. Why are we doing a block transform at all ? It's not obvious. We want to exploit correlation. But there are plenty of other ways to do that. We could use a DPCM method (delta from neighbors), or just a probability-model method (predict similarity to neighbors). That could capture the correlation perfectly well. So it's not that.

Sometimes it's said that it's good for quantization. Hmm, maybe. You can also quantize the error that you would code in a DPCM method (like lossy CALIC). In practice quantizing transforms does work better at high loss, though it's a bit mysterious why that is exactly. There are two main factors I think :

1. Separating the DC and the AC. People make a lot of claims about the DCT basis being a natural fit for the human visual system. I think that's largely hogwash. Certainly once you cut it into 8x8 blocks it's wrong. But one thing that is valuable is the seperation of DC (low-frequency intesity - basically just a mipped-down version of the signal), and AC (higher frequency detail). The transform lets you code the DC carefully and throw away the AC.

Now, you could say making a lower res mip version and sending that first is not really inherent to transform coding. You'd be wrong. The problem is you want to make a lower res "DC" version without introducing redundancy. Say you have a 2x2 block of pixels -

| a b |
| c d |

You want to send a lower res version first, such as (a+b+c+d)/4 , and then send the higher res version. Say you want do this with some kind of DPCM/predictor scheme. Even if you use the lower res to predict the higher res, you are creating redundancy, you're now coding 5 values instead of 4. Of course the way to fix this is to do a 2d Haar transform to create the average & deltas from average in a reversible way that only makes 4 values instead of 5 !!

(note you could just send {a} as the low res version then send {b,c,d} later - that avoids redundancy at the cost of not having as good of a low res version).

2. Efficiency (lots of zeros). Block transforms are handy in practice just for a purely practical reason. When you quantize them you get lots of zeros. That lets you do run-length or end-of-block codes that cut off a ton of zeros and save lots of coding ops. Note that I'm distinguishing this from the "energy compaction" which they often talk about in the literature as being a huge win for compression. No, that's not really why we like block transforms - you can get the exact same entropy gain by using the correlation in other ways. The big win is the practical issue.

05-07-09 | Integer Function Inversions

Question : how do you find/make functions that are exactly invertable in integers?

In particular, what I'd like is a family of cumulative probability distribution functions for arithmetic coding that can be inverted in integers.

Even more specifically, the main cases are semi-laplacian or semi-gaussian functions. Precisely the goal is something like this :

Probably of a symbol is modeled like P(x) ~= e^ ( - lambda * x ) = k ^ -x

(or something like that; P(0) is large, P(255) is small; x in [0,255] and)

Cumulative probability is the sum of probability of all lower symbols :

C(x) = Sum { y <= x } P(y)

To encode x we send something in [ C(x-1) , C(x) )

We want C(x) to be scaled such that C(255) = 16384 or some other power of 2 constant

We want C(x) to be integers, and we for decodability we must have C(x) >= C(x-1) + 1

That is, the integer math truncation must never make a C(x) = C(x-1)

Now, to decode we get back some target number T that's in in [ C(x-1) , C(x) ) for some X

We'd like to have an analytic function that gives us x directly from T :

x = D( T )

Now before you say "that's impossible" ; it's obviously not. You can certainly trivially find solutions such as :

C(x) = (C(255)/256) * (x+1)

D(T) = 256 * T / C(255);

The question is, are there more useful solutions, and in particular can you construct a whole family of solutions that are parameterized to give you different shapes.

Obviously you can precompute table lookups, but I'd rather not have a whole mess of tables.

I don't really know anything about integer->integer function theory; it seems to me there are two possible approaches to this. One is "constructive" ; start with simple funcionts that you know you can invert, then there are many operations you can do on them to compose or distort them and still have them invertable.

05-05-09 | Infinity

So there's this game Infinity which I guess has been in development since 1950 or something. Recently the tech post on Deferred lighting was linked around the blogosphere. It led me to this page, and there's some interesting stuff on it.

Back in 1999 or so I started working on my "Galaxy" codebase. My original goal was to develop my own VIPM codebase so that I could use it to make a game where you fly around a galaxy. I wanted to explore continuous scale zooming, to be able to fly from one star system to another, and then right down to planet surfaces. I conjectured that with the current hardware (TNT2) and VIPM technology I could do it and it would be amazing.

It turns out "Infinity" is roughly the same vision. I guess it's not that surprising, it's a common dream. I wanted to make a game kind of like "Trade Wars". I guess "Privateer" also was in dev around that time and was the kind of thing I wanted too, thought it would up sucking. Of course eventually Eve Online came out and did a lot of the things I wanted, but not in the way I wanted - I wanted more of a solo action game, not a team-oriented politics game. I wanted to actually fly my spaceship like in Wing Commander or whatever.

Part of why I wanted to make a space game in 1999 was that I thought the rendering and realism would be a lot easier. You don't have to draw humans or other soft bodies or hair or any of that kind of stuff that we don't do well. You just have lots of shiny metal and such. Everything is rigid bodies so you don't need a fancy animation system. Most of the art would be procedural so I could generate it from code and not have to hire artists except for the ships.

There's a lot of fun rendering and visual stuff you could play with in a Galaxy game; I always wanted to do atmospheric scattering - not just when you're on the surface, but from any altitude, so it would be really volumetric, with the varying density of atmosphere and different particulate composition as you get higher into space (and stuff you would see as the sun(s) go behind other planets and the light passes through the atmosphere around them). You could do fun nebulas and planet's rings, gases that speed up and radiate as they fly around a black hole, etc.

Anyway, I wound up never actually making the game or getting into the real space game technology (you have issues with coordinate precision for example; you need to keep absolute coordinates of everything in doubles and then transform your objects to be camera-relative for render).

The Infinity dev diary describes some good stuff. The terrain heightmap is procedural from noise functions that are evaluated on the GPU to make tiles. Texturing is with a kind of "splat" or "triblend" system; procedural blending by height & slope to select various patterns.

I wonder if you could now make a game that was "always on". Like the Galaxy/Infinity space game, and if you're not logged in to the MMO, time keeps ticking and your ship is still there. If it was an iPhone game or something, you always have your phone on you so you can keep logging back in all the time. You could do things like build your own starbases for trading, and if someone attacks one, it would ring your phone to let you know you should log in. Obviously this would be a horrible thing to do to people but also very compelling.

05-05-09 | AutoReflect

I wrote before about autogenerating prefs for C++ . Well, I went and did it.

It's almost exactly what was discussed before. There's a code generator "AutoReflect" that makes a little include file. You mark variables with //$ to get them processed.

Here's an actual example from Galaxy4 :

class DriverTestPref : public Prefs
public :


    float m_sharedConvergeTime; //$ = 2.2f;
    float m_cubicMaxAccelScale; //$ = 1.75f;
    float m_pdTimeScale; //$ = 2.f;
    float m_pdDamping; //$ = 1.f;
    float m_pdMinVel; //$ = 0.005f;
    float m_interceptConfidence; //$ = 0.5f;
    //float m_test; //$ = 1.f;

#include "gApp_DriverTest.aup"

To integrate this with VC 2003, I made AutoReflect just run as a global pre build step. It recurses directories and looks for all the ".aup" files. It checks their modtime against the corresponding .cpp and only runs if the cpp is newer. Even then, it's quite common that the cpp was changed but not in a way that affects the AutoReflect, so I generate the new aup file to a temp name, diff it against the current aup file, and don't touch it if it's the same.

That way I don't have to worry about adding custom build steps to lots of files or anything, it's one global pre-build you put in your project once. There are a few minor disadvantages with that :

1. You have to make an "aup" file manually once to get it started. You can do this just by creating the file by hand, or you can run "AutoReflect -f" for "full process" in which case it changes the enumeration to look for all "cpp" files instead of looking forb all "aup" files.

2. Fucking MSDev Pre/Post Build Events don't use the machine %PATH% to look for executables !?!?! URG WTF. It means I can't just put "AutoReflect" in there and have it work on various systems, I have to hard code the full path, or put it in one of the MSDev "Executable Directories" paths.

I gather than in VC 2008 the custom build capabilities are much enhanced so maybe there's a better way there, but this is mostly working very nicely. One good thing about doing it as a pre-build is that it doesn't interfere with the normal Make incremental build at all. That is, when the cpp is modified, first I make a new aup, then the cpp is compiled (which includes the new aup). There's no funny business where the cpp gets compiled twice, or it gets compiled before the aup is made or any of those kinds of problems.

ADDENDUM : actually I just realized there is a problem with this method. Because the "pre build" is only run for an F7 "build" and not for a ctrl-F7 "compile" you can compile your file and it doesn't get the new AUP. That's not a disaster, but it mildly sucks, I'd like it to AutoReflect before the compile when I hit ctrl-F7.

For the example above, the actual "aup" generated is :

template <class T>
void DriverTestPref::Auto_Reflection(T & functor)

void DriverTestPref::Auto_SetDefaults()
    m_sharedConvergeTime = 2.2f;
    m_cubicMaxAccelScale = 1.75f;
    m_pdTimeScale = 2.f;
    m_pdDamping = 1.f;
    m_pdMinVel = 0.005f;
    m_interceptConfidence = 0.5f;

While I was at it I also put my Prefs & TweakVars into a "DirChangeWatcher" so that I get automatic hot reloads and made that all happen by default in Galaxy4. Pleasing.

I plan to not check in the aups to source control. Since they are generated each time you build, I'll treat them like obj's. Again the only problem with this is when someone syncs and doesn't have the aups yet - I can't do my incremental build method until they exist. What I would really like is for the MSDev "Full Rebuild" or "Clean" to run my "AutoReflect -f" for me that would generate the aups.

There's one stupid thing that's still not done in this, which is handling .h vs .cpp ; since you can have autoreflected classes in xxx.h and xxx.cpp , both would generate xxx.aup and I'd have to merge them or something. I could make it generate two seperates aups, "xxx.h.aup" and "xxx.cpp.aup" , not sure if that's the right thing to do. (ADDENDUM : yeah, I just did that, I think it's the way to go, it also makes it work with .c or whatever, because for any .aup file I can find the source file by just cutting off the .aup ; it removes all assumptions about the source file extension).

Of course I talk about AutoReflect mainly in terms of "prefs", but it's useful for other things. It basically gives you reflection in C++. One thing I'd like to use it for is to bring back the "IOZ" automatic IO system we did at Oddworld (basically a templated visitor IO that let's you stream things in and out trivially).

Unofficial early releases :

AutoReflect.zip (zip 62k)

cblib.zip (zip 500k)

galaxy4.zip (zip 1.5M)

Also in galaxy4 :

Now on Dx9. Now shares math/core code with cblib so it's not duped. New OBB & Hull code as written about earlier (in gApp_HullTest). New SmoothDriver and test app (gApp_DriverTest) (cubic & pd controller stuff written about long ago). Some other random shit, like new gFont,

05-04-09 | Cock Ass Shit

God damn all you drivers on your cell phones with your giant fucking Suburbans and whatnot that I can't see around, and when you come right into my lane I know you wouldn't even feel me in an accident so I just have to make evasive maneuvers.

God damn all you rubberneckers. Just because there's a fucking cop stopped on the opposite side of the freeway doesn't mean you need to slam on your breaks and look over there. JUST FUCKING GO. Your job when are your driving is to rapidly vacate the space you are in so someone else can use it. Get the fuck on. God damn all you slow movers leaving giant gaps. When you leave a huge gap in front of you, people just keep tucking in to it, and that slows down every single person behind you. It's not curteous, it's fucking rude to the entire line of cars in your lane behind you. God damn all you people who don't accelerate after getting past a jam up. It's your duty to get the flow moving again. When you get through a constriction, like a 3->2 lane reduction, it's your duty to step on it to create a pressure drop to pull the people behind you in more quickly.

God damn you people who don't park up against the edge of the possible parking area, so that we only get two cars on a curb that should fit three. God damn you people who pull up to a red light on a street with two lanes and block the right lane when the left was open, preventing me from turning right on red.

God damn KEXP for playing fucking WoPop and Shake the Shack and all that weirdo awful music right through the drive time rush hour. God damn Seattle for having all the good game companies on the east side and all the good hipsters on the west side. God damn all you motherfuckers cheating in the carpool lane; yes I see you, and yes it's a 3 person carpool lane, 2 does not cut it. God damn all you onramp users who drive as far as possible forward in the carpool lane and then jam yourself into the line unsmoothly, resulting in blocking the carpool lane and causing a big fracas in the right lane because you didn't zipper where you had a good opening.

God damn ING Direct for asking me so many fucking questions every time I log in. I can't remember them all so I have to write them down on a piece of paper next to my monitor. Great fucking security system, thanks a lot for making my transaction more secure. In fact it is great for them because they have pushed the fault to me and can claim that they did everything they should.

There's a ton of RPG's and Fantasy RPG/RTS games I never heard of :

King's Bounty: The Legend
Drakensang: The Dark Eye
Sacred 2: Fallen Angel
Mount & Blade
Elven Legacy
Kohan II: Kings of War

I hate the feeling of being a slave to the traffic schedule. I hate waking up in the morning around 8 and knowing I can't leave for work until after 10. I'm ready to go! Fuck. Then around noon each day at work it hits me that I'm gonna be stuck until 7 or 7:30. Often that's fine, I'm gonna work until then anyway, but just knowing that I'm fucking stuck and don't have my liberty gives me a flash of panic and frustration. I just want to work until my brain is exhausted and then go home.

I have the same kind of feeling at home with the noisy neighbors. They haven't really even been that bad recently, but every time I hear a bit of noise I worry if it's just a precursor to a storm. Even if nothing happens it makes me tense and worried. Every day if I think about going to sleep early I wonder if it will be okay or if I should stay up and wait for them to go to sleep first.

Here's a puzzle for you : WTF is a Cafe Creme ? The French style espresso with cream; I mean, I know what it is, but what *exactly* ? How would I get one in the US ? It's not exactly just an espresso with cream added; maybe the cream is foamed or something; I think the shots are long pulls too but can't say for sure.

The Julie and Julia movie literally makes me sick to my stomach. I adore Julia Child much like I adore Jacques Pepin. She (Julia) was a nut, someone completely full of life, an independent spirit, a trailblazer. This fucking Julie whore is some stupid whiney blogger. Get the fuck out of my movie about Julia Child. How dare you even be mentioned in the same breath. I'm revolted by the glorification of bloggers. Look I like reading blogs as much as the next guy, but I know perfectly well it's akin to reading tabloids. It's trash, it's insignificant.

The revolting trend of food bloggers who think way too much of themselves makes me want to just check out of life so I that I don't have to be connected to them in any way. I'm sick of reading fucking Michael Ruhlman constantly blogging about the media ventures with other bloggers or the success of his fucking book. I'm sick of the GastroGnome name dropping about what fucking special invite-only food event she got invited to. It's fucking self-indulgent self-aggrandizing gossip. Write about fucking food and that's all. If you want to write a cookbook that's about the recipes, fine fine. But don't write a fucking book about being a food blogger! Be more like Dave Lebovitz .

05-03-09 | Smallpox

The NYT today had a pretty severe misrepresentation of the truth. In the chart of infectuous diseases you see the typical big killers, like the 1918 Spanish Flu, but then you see Smallpox listed in 1947 with three cases.


I guess that's sort of true, but the actual number for 1947 in NYC was 12 cases. In 1947 Smallpox was well gone from the US. It was still killing in the rest of the world. Suddenly 3 cases appeared; the government quickly enacted a vaccination program and quarantined those people; the total infected reached 12 but the spread of the virus was fully controlled.

The point is it's bizarre to list that particular 1947 small outbreak in NYC as "smallpox" on the chart (everything else on the chart lists worldwide effects). It leads you to think smallpox wasn't a big deal. Au contraire. In the first half of the 20th century, smallpox was still a virulent killer. In fact, it was so common that you would hardly even say it was an "epidemic", it was just constant; hundreds of thousands of people died from it every single year (sort of like Malaria still is) (I guess that's called "endemic").

In fact you can quote a better NYT article on smallpox :

Smallpox killed more people over the ages than any other infectious disease. In the 20th century alone, experts estimate, it took up to a half billion lives, more than all the wars and epidemics put together.

(I'm guessing most of that was in the 3rd world (?) ; the Smallpox vaccine was invented around 1800, but it wasn't eradicated until 1977 ; I haven't seen good numbers on when widespread vaccination was adopted in the 1st world)

The Wikipedia on Smallpox also has this nasty number :

The disease killed an estimated 400,000 Europeans each year during the 18th century

That's 40M in the century, which is a hell of a lot when you consider the low population of Europe in the 18th century. (population of Europe was 100M for almost all of the 18th century, then shot up to 200M near the end with the Industrial Revolution and the growth of cities).

Smallpox is also extremely important historically. The first vaccination was smallpox. (in fact the term vaccination comes from vache or vaca for cow - it was a cowpox , a variant of smallpox, injected as a proxy virus that was less deadly and built the right antibodies).

Smallpox was also the secret weapon of colonists. It's the primary way that Cortez was able to defeat the Aztecs.

05-02-09 | Are you fucking kidding me ?

I finally got my HTPC working with S3 sleep. It's pretty easy actually, you just set it in the BIOS, and then there's a Windows setting you have to do. I kept it simple and just disabled all USB-wake options, which can apparently cause some problems.

For those out of the loop S3 Sleep for desktops is just like the good way that laptops have been sleeping forever - it turns off everything except your RAM. The normal S1 Sleep for desktops is pretty retarded because fans and disks and such keep running; what's the point of that?

Anyway, everything was working fine and I was all happy, until I noticed my volume control stops working after a sleep. So I went to the M-Audio web page and found :

Sleep (Hibernation) Mode
Text size [-] [+]

Q: Can I use sleep or hibernation mode with my M-Audio device?

A: M-Audio does not support sleep mode with any of its devices. The ability to re-establish a connection with a 3rd party device driver after waking from sleep or hibernation mode is system dependent. Because of this, M-Audio recommends setting your computer to never go to sleep. Instead, it is recommended that you turn off your monitor, or completely power off the computer. If your M-Audio device is not working after waking from sleep mode, restart your computer to re-establish a connection with the device.

!? WTF !? Friggle frack fucking douchebag morons. It just blows my mind how so many people think it's okay to be like "oh yeah our stuff is totally broken, that's a known feature and we're not going to fix it". WTF. How can you sleep at night? (See also 1 , 2 , 3 )

In better news on the HTPC front, I finally got the AMD Cool & Quiet working, and it's actually pretty awesome. You have to install the latest AMD Processor Driver, and then set your power scheme to "Minimal" and it automatically kicks in. You can also run the AMD Processor Monitor to watch your MHz go up and down.

That on its own doesn't do a whole lot for you, but I also got the variable fan speeds working. I'm running two 120 mm Scythe quiet fans that are supposed to run super slow. I had one of them plugged in the "NB Fan" power plug, and I had another plugged directly to the power supply, dunno why I did that. Turns out on my Gigabyte mobo, it does fan speed stepping if you plug them in to "CPU Fan" and "SYS Fan" power plugs on the mobo. Now both of them are running super slow all the time and the whole thing is super quiet.

I still have a slight whir from the machine, which I think is the PSU fan, so some day I may have to replace that. I also noticed while I was in there that the whole machine is absolutely jammed full of dust even though its only 6 months old. A computer sitting in a living room with fans running all the time is basically an air filter, it's sucking all the dust in the room right through its central cavity. Then it gets stuck in the fans and they get loud.

... oh and I also finally took care of the damn "alt key getting stuck" problem. I just wrote a program to look for alt being down and to send an alt-up-down itself. It just uses GetAsyncKeyState to check for keys being down then uses SendInput to toggle them. BTW while I was in there I discovered VkKeyScan, MapVirtualKey, and ToAscii which I somehow didn't know about and can be useful for eg. converting between ascii and VK codes.

In other news I was thinking about playing another action-RPG video game. I can't run anything newer than like 2005 or so, nor can I run any console games. I tried Morrowind a while ago, but the graphics just look like ass to me. I really find 3d unbearably ugly 99% of the time, I'd much rather have painted backgrounds. I've played all the old Icewind Dale and Baldur's Gate games; ideally I'd find something like that.

I might try Dungeon Siege again; I played it when it came out and thought it was ass. Also there's this new one "Titan Quest" but it might push the graphics too hard for my old lappy.

05-01-09 | Bad Board Games

Kim linked this article about some fruity pants board games . I think it's a great article, because I think it's a fantastic example of totally retarded wrong-headed board game design.

I've been going to board game night recently here, and it's been rather hit or miss. There are some good interesting games, but a lot of stinkers. (BTW I classify board games & card games as pretty much the same thing).

Good board games create a dynamic system where you can make real strategic choices. That is, you should have decision points where there is not simply one provably best move. There should be multiple choices that allow you to play in different styles.

The most interesting thing in real board games is when there's interaction between the players; playing against the rule system is not interesting, it's always the same, but playing against other people adds variety and lots of metagame; the metagame of the human interaction outside the rule system is perhaps the most interesting part of board games. Really retarded board games like Monopoly or Risk can only be saved through the metagame. A lot of bad board games basically have you playing against yourself, to try to get as high a score as possible, other people are playing too but you don't really interact much with each other.

Conceptual games like "Train" are some of the worst. There are a lot of them, if you go into any board game shop probably half of them are what I would call "conceptual". That is, the whole point of these games is basically just the idea of them - the setting, oh aren't the characters cute, oh this one is about killing kittens to harvest their souls, what a cute idea. The actual play of these games is trivial and uninteresting. If you want a freaking story, read a book. This is not an appropriate use of the board game medium.

Rather than having a moment of "revelation" in conceptual games, I usually have a moment of abject depression and futility when I realize that the game I'm playing is totally pointless and retarded and I'm just rolling dice and flipping cards for no good reason. Someone could've just told me the concept in one sentence and then we wouldn't have had to waste our time playing this damn game.

Which brings me to the next type of game that sucks - games where all the difficulty and "skill" just come from overly complex rules. For one thing, it sucks playing these games for the first time because you spend hours trying to learn them. Then once you learn the basic rules it takes a little while to really learn what the strategy dynamic is. Then once you understand the strategy, you realize the game is totally trivial. If you have opponents who actually know how to play, then the game is random. Once you get to the level of understanding the game system, then these games have no more complexity at all. There are actually some games that are widely considered to be good games that fall into this category, such as Settlers or Domaine. Those games can be saved by playing in a group and relying on the human metagame to make them more interesting, but basically they are just very complex strategic systems that don't have real depth. Perhaps the most extreme example is the old game Axis & Allies where the static initial layout made it so that once you were an expert your entire move was predetermined and the game just became a giant dice toss. Random board setups prevent the complete destruction of a game in that way, but don't actually make the strategic system any more interesting.

The test for whether a board game is actually really good goes something like this : 1. convert all the pieces to abstract solid colored blocks, convert the board to just squares with no graphics. 2. teach all the players the strategy so they know exactly what the right way to play is. We've reduced the appeal of the game to only it's rule system, and we've removed all the difficulty due only to unfamiliarity or over-complex rules. Is it still interesting to play ? If not, then it's basically a garbage game dressed up in a pretty outfit. The truly great games are still interesting in this format (chess, go, poker, diplomacy) but most are shit.

A really good game (chess, go, poker) is a beautiful thing. It can teach you about yourself and the universe. As you get better and better, you keep reaching plateaus where you see something new in the strategy system that you didn't even know was there before. You can play opponents who use some weird style that you're not familiar with that can teach you new things about the game.

04-30-09 | Sex Roles

I certainly don't romanticize the historical oppression of women (neither the near-slavery of long ago or the social repression of 1950's America). However, there is something romantic and appealing about complementarity. Complementarity is the idea that you and your lover should be very different, have different skills, and together make a whole. It makes you value the other person, it makes you need them, and it lets you be better as a whole than you could be as individuals, because you can specialize and spend more time in each of your abilities. Plus there are many personality traits that are very useful but hard to have at the same time. For example, one of you could be very intellectual and rational and serious, the other could be very emotional and carefree. Those are both wonderful useful traits, but if you try to combine them you just sort of water them down and make them worse. One of you could be tough and demanding and aggressive, the other could be sweet and friendly.

In the past lots of people were forced into marriage by their sheer inability to survive alone. Women would have a hell of a hard time supporting themselves, and the men were widely incapable of doing basic things like dressing themselves or making a sandwich. That's a very strong bond; you see it still with old couples who despise each other, but have to stick together because the man doesn't know how to run a washing machine and the woman doesn't know how to drive.

Of course I think of people now as being pretty independent, but a lot of them are still totally incompetent. I'm amazed how many men still can't do basic cleaning or cooking (I'm sure they quite willfully keep themselves ignorant because they don't want to have to do that stuff). Lots of girls still couldn't get a decent job if they had to, because they were Communications majors or something and have zero skills. And of course all those suburban princess girls are just completely worthless all around, they can neither cook nor clean nor do any decent work. They only thing they know how to do is put on make up, wait tables, and give blow jobs.

Relationships without artificial glue are very difficult to sustain. As I've already mentioned, one form of strong glue is if you're incompetent in some way and simply can't live apart (or societal repression as still exists in much of the world). Another form of strong glue is children. Even if you aren't fully in the mode of "we hate each other but we'll stay together for the children", children are still strong glue in that it simply gives you something that you both care about and have to take care of all the time; it gives you a common mission, and it also gives you a focus of grief that's outside of each other.

Another form of strong glue is being a loser. By being a loser I mean the fear of being single or the belief that you can't do better, or inexperience dating. If you marry your highschool sweetheart, you're terrified of having to get back out there and date. You believe maybe this is the only person you can ever find, you're afraid of what single life would be like. That's a strong glue.

Modern society celebrates independence and consciously choosing a mate - that is, being in a relationship without all that glue. But with none of that glue your only reason to be together is because you think the life with this person is better than any other life you could have, and that is a very hard bar to meet all the time.

So, my advice to someone who wants to have a strong relationship : 1. make yourself incompetent, if you're a man never learn how to cook or clean, have your mom always pick your outfits for you. 2. marry the first girl you date or have sex. 3. have kids right away. 4. don't let your wife go to college or learn any skills.

04-29-09 | QCD

Someone made me think briefly about QCD (Quantum Chromo Dynamics).

I never really got my head around the standard model in grad school. I think I understood QED pretty well, and Weak isn't really that bad either, but then you get into QCD and the maths gets really tough and there's this sea of particles and I had no idea what's going on. Part of the problem is that a lot of the texts go through a historical perspective and teach you the stages of understanding and the experiments that led to modern QCD. I think that's a big mistake and I discourage anyone from reading that. I was always really confused by all the talk of the various mesons and baryons. Often the classes would start with talking about K+ transitions or pion decay or scattering coefficients for "Omegas" and I'd be like "WTF are these particles and who cares what they do?".

I think it's way better just to say "we have quarks and gluons". And yes, the quarks can combine together into these various things, but we don't even really need to talk about them because nobody fucking cares about what exactly the meson made from (strange-antistrange) is called.

I much prefer a purely modern approach to QFT based on symmetry. In particular I really like Weinberg's approach in his textbook which is basically - we expect to observe every phenomenon in the universe which is *possible* to exist. If something is possible but doesn't ever happen, that is quite strange and we should wonder why. In particular with QFT - every possible Lagrangian which leads to a consistent theory should correspond to something in nature. When you start to write these down it turns out that very few are actually possible (given a few constraints, such as the postulate that relativity is required, etc.).

Anyway, I was never really happy with my intuition for QFT. Part of the problem is the math is just so hard, you can't do a lot of problems and get really comfortable with it. (David Politzer at Caltech once gave me a standard model homework problem to actually compute some real scattering coefficients that had been experimentally tested. It took me about 50 pages and I got it horribly wrong).

The whole gauge-field symmetry-group idea seems like it should be very elegant and lead to some intuition, but I just don't see it. You can say hand wavey things, like : electromagnetism is the presence of an extra U(1) symmetry; you can think of this as an extra circular dimension that's rolled up tiny so it has no spatial size, or if you like you can do the Feynman way and say that everything flying around is a clock that is pointing in some direction (that's the U(1) angle). In this picture, the coupling of a "charge" to the field is the fact that the charge distorts the U(1) dimension. If you're familiar with the idea of general relativity where masses distort spacetime and thus create the gravity force, it's the same sort of thing, but instead of distorting spacetime, charge distorts the U(1) fiber. As charges move around in this higher-D space, if they are pushed by variation of the U(1) fiber clock angle, that pushes them in real space, which is how they get force. Charges are a pole in the curvature of the fiber angle; in a spacetime sense it's a pinched spot that can't be worked out by any stretching of the space fabric. Okay this is sort of giving us a picture, but it's super hand wavey and sort of wrong, and it's hard to reconcile with the real maths.

Anyway, the thing I wanted to write about QCD is the real problem of non-perturbative analysis.

When you're taught QED, the thing people latch onto are the simple Feynman diagrams where two electrons fly along and exchange a photon. This is appealingly classical and easy to understand. The problem is, it's sort of a lie. For one thing, the idea that the photon is "thrown" between the electrons and thus exchanges momentum and forces them apart is a very appealing picture, but kind of wrong, since the photon can actually have negative momentum (eg. for an electron and positron, the photon exchanged between them pulls them together, so the sort of spacemen playing catch kind of picture just doesn't work).

First of all, let's back up a bit. QFT is formulated using the sum of all complex exponential actions mechanism. Classically this would reduce to "least action" paths, which is equivalent to Lagragian classical mechanics. There's a great book which teaches ordinary Quantum Mechanics using this formulation : Quantum Mechanics and Path Integrals by Feynman & Hibbs (this is a serious textbook for physics undergrads who already know standard QM ; it's a great bridge from standard QM to QFT, because it introduces the sum-on-action formalism in the more familiar old QM). Anyway, the math winds up as a sum of all possible ways for a given interaction to happen. The Feynman diagram is a nice way to write down these various ways and then you still integrate over all possible ways each diagram can happen.

Now let's go back to the simple QED diagram that I mentioned. This is often shown as your first diagram, and you can do the integral easily, and you get a nice answer that's simple and cute. But what happened? We're supposed to sum on *all* ways that the interaction can happen, and we only did one. In fact, there are tons of other possibilities that produce the same outcome, and we really need to either sum them all, or show that they are small.

One thing we need to add is all the ways that you can add vacuum -> vacuum graphs. You can make side graphs that start from nothing, particles pop out of the vacuum, interact, then go back to the vacuum. These are conveniently not mentioned because if you add them all up they have an infinite contribution, which would freak out early students. Fortunately we have the renormalization mechanism that sweeps this under the rug just fine, but it's quite complex.

The other issue is that you can add more and more complex graphs; instead of just one photon exchange, what about two? The more complex graphs have higher powers of the coupling constant (e in this case). If the coupling constant is small, this is like a Taylor expansion, each term is higher powers of e, and e is small, so we can just go up to 3rd order accuracy or whatever we want. The problem with this is that even when e is small, as the graphs get more complex there are *more* of them. As you allow more couplings, there are more and more ways to make a graph of N couplings. In order for this kind of Taylor expansion to be right, the number of graphs must go up more slowly than 1/e. Again it's quite complex to prove that.

Starting with a simple problem that we can solve exactly, and then adding terms that make us progressively more accurate is the standard modus operandi in physics. Usually the full system is too hard to solve analytically, and too hard to get intuition for, so we rely on what's called a perturbation expansion. Take your complex system that you can't solve, and expand it into Simple + C * Complex1 + C^2 * Complex2 + ... - higher and higher powers of C, which should be small.

And with QCD we get a real problem. Again you can start with a simple graph of quarks flying along passing gluons. First of all, unlike photons, there are gluon-gluon couplings which means we need to add a bunch more graphs where gluons interact with other gluons. Now when we start adding these higher order terms, we have a problem. In QCD, the coupling constant is not small enough, and the number of graphs that are possible for each order of the coupling constant is too high - the more complex terms are not less important. In fact in some cases, they're *more* important than the simpler terms.

This makes QCD unlike any other field theory. Our sort of classical intuition of particles flying around exchanging bosons completely breaks down. Instead the quarks live in a foaming soup of gluons. I don't really even want to describe it in hand wavey terms like that because any kind of picture you might have like that is going to be wrong and misleading. Even the most basic of QCD problems is too hard to do analytically; in practice people do "lattice QCD" numerical computations (in some simple cases you can do the summations analytically and then take the limit of the lattice size going to zero).

The result is that even when I was doing QFT I never really understood QCD.

04-28-09 | Quadratic

I'm doing a little refinement of my old cubic interpolator ("Smooth Driver") thing. (see also : here and here and here ).

One thing I'm trying to do is fix up all the nasty epsilon robustness issues. A small part of that is solving a quadratic. "Easy!" I hear you say. Everyone knows how to solve a quadratic, right? Not so.

I found this page which has a nice summary of the issues, written by a sour old curmudgeon who just whines about how retarded we all are but doesn't actually provide us with a solution.

You can also find the Wikipedia page or the Numerical Recipes (5.6) snippet about the more robust numerical way to find the roots that avoids subtracting two nearly identical numbers. Okay, that's all well and good but there's a lot more code to write to deal with all the degenerate cases.

This is what I have so far : (I'm providing the case where the coefficients are real but the solutions may be complex; you can obviously modify to complex coefficients or only real solutions)

// A t^2 + B t + C = 0;
// returns number of solutions
int SolveQuadratic(const double A,const double B,const double C,
                    ComplexDouble * pT0,ComplexDouble * pT1)
    // first invalidate :
    *pT0 = FLT_MAX;
    *pT1 = FLT_MAX;
    if ( A == 0.0 )
        if ( B == 0.0 )
            if ( C == 0.0 )
                // degenerate - any value of t is a solution
                *pT0 = 0.0;
                *pT1 = 0.0;
                return -1;
                // no solution
                return 0;
        double t = - C / B;
        *pT0 = t;
        *pT1 = t;
        return 1;
    else if ( B == 0.0 )
        if ( C == 0.0 )
            // A t^2 = 0;
            *pT0 = 0.0;
            *pT1 = 0.0;
            return 1;
        // B is 0 but A isn't
        double discriminant = -C / A;
        ComplexDouble t = ComplexSqrt(discriminant);
        *pT0 = t;
        *pT1 = - t;
        return 2;
    else if ( C == 0.0 )
        // A and B are not zero
        // t = 0 is one solution
        *pT0 = 0.0;
        // A t + B = 0;
        *pT1 = -B / A;
        return 2;

    // Numerical Recipes 5.6 : 

    double discriminant = ( B*B - 4.0 * A * C );
    if ( discriminant == 0.0 )
        double t = - 0.5 * B / A;
        *pT0 = t;
        *pT1 = t;
        return 1;
    ComplexDouble sqrtpart = ComplexSqrt( discriminant );
    sqrtpart *= - 0.5 * fsign(B);
    ComplexDouble Q = sqrtpart + (- 0.5 * B);
    // Q cannot be zero
    *pT0 = Q / A;
    *pT1 = C / Q;
    return 2;

One thing that is missing is refinement of roots by Newton-Raphson. The roots computed this way can still have large error, but gradient descent can improve that.

04-27-09 | Getting Old

Last weekend we went to the arboretum, and I was being retarded as usual and jumping around. I jumped right in a mud puddle and slipped and landed with my knee on a rock. It hurt a bit at the time, not a huge deal, but my knee has been bruised and swollen for the whole last week. Even minor injuries seem to last so long and be so crippling now.

I can't stand the music the kids listen to these days. There have always been youthy genres I've hated (like rap-rock and all the mainstream pop), but one that really blows my mind these days is the "pop-punk" that the kiddies seem to love. Stuff like : crap or ass or dear god .

I think all the piercings and tattoos and such are ugly, unsanitary, and completely non-unique and not rebellious. Kids seem to love it.

I think twitter and facebook and myspace and cell phones and texting and all that is just an awful waste of time that never conveys any real information or interesting conversation. It's just a masturbaturoy attempt to reassure each other and feel connected, but it's not a real connection to anything natural.

I still get a paper newspaper, think that blogs are an awful place to get news, and that reading on monitors is extremely unpleasant.

Girls my age just look really old to me, all wrinky and saggy and dried up, like a bunch of bones flopping around inside an uninflated balloon. But girls that are young seem like retarded aliens.

I think text-message abbreviations should be used only when texting, and even then used as little as possible. I think eloquent speech and good grammar and to be strived for at all times; though I often fail, I would never write "srsly" or "imo" unless I was being ironic.

I'm getting old and it fucking sucks.

04-24-09 | Convex Hulls and OBB's

I wrote last month a bit about OBB fitting. I mentioned at the time that it would be nice to have an implementation of the exact optimal OBB code, and also the bounded-best OBB in reasonable time. I found the Barequet & Har-Peled work on this topic but didn't read it at the time.

Well, I finally got through it. Their paper is pretty ugly. Let me briefly explain their method :

They, like my old OBB stuff, take heavy advantage of the fast rotating-calipers method to find the optimal rectangle of a convex hull in 2d (it's O(n)). Also finding the convex hull is O(nlogn). What that means is, given one axis of an OBB, you can find the optimal other two axes in O(nlogn). So the problem just comes down to finding one of the optimal axes.

Now, as I mentioned before, the number of actual axes you must consider to be truly optimal is O(n^2) , making your total run time O(n^3). These axes are the face normals of the convex hull, plus axes where the OBB is supported by two edges and a vert (I don't know an easy way to even enumerate these).

It's now pretty well known around the industry that you can get a very good OBB by just trying a bunch of scattered initial axes instead of doing O(n^2). If you try some fixed number of axes, like say 256, it doesn't count against your big-O at all, so your whole OBB fit is still O(nlogn).

Well this is exactly what the Barequet & Har-Peled method is. They try a fixed number of directions for the seed axis, then do the rectangle fit for the other two axes. The main contribution of the paper is the proof that if you try enough fixed directions, you can get the error of the bbox within whatever tolerance you want. That's sort of intuitively obvious - if you try more and more fixed directions you must get closer and closer to the optimal box. Their construction also provides a specific method for enumerating enough directions.

Their enumeration goes like this :

Start with some seed box S. They use the box that is made from taking one axis to be the "diameter" of the point set (the vector between the two most seperated points). Using that box is important to their proof, but I don't think which seed box you use is actually terribly important in practice.

The seed box S has normalized edge vectors S.x , S.y, S.z (the three axes that define the box).

Enumerate all sets of 3 (non-negative) integers whose sum is <= K , that is {i,j,k} such that (i+j+k) <= K

Construct the normal N = i * S.x + j * S.y + k * S.z ; normalize it, and use this as the direction to fit a new OBB. (note that there are a lot of points {ijk} that generate the same normal - any points that are integer multiples of another; those can be skipped).

Barequet & Har-Peled prove that the OBB made this way is within (1/K) to some power or other of optimal, so as you increase K you get ever closer.

Now, this is almost identical to my old "OptimalOBBFixedDirections" which tried various static directions and then optimized the box from there. My OptimalOBBFixedDirections always tests 42 directions in the {+,+,+} octant which I made by subdividing an octahedron. I have found that the Barequet & Har-Peled method does in fact find better boxes with fewer tests, but the difference is very very small (thousandths of a percent). I'll show numbers in a second.

First I want to mention two other things.

1. There's a "common wisdom" around that net that while it is bad to make an OBB from the covariance matrix of the *points* , it is good to make an OBB from the covariance matrix of the *volume*. That is, they claim if you have a closed mesh, you can use the Mirtich (or Eberly or Blow/Binstock) method to compute the covariance matrix of the solid body, and use that for your OBB axes.

I have seen no evidence that this is true. Yes, the covariance matrix of the points is highly dependent on the tesselation, while the covariance matrix of the solid is more a property of the actually shape of the object, so that is intuitively pleasing. In practice it appears to be completely random which one is actually better. And you just shouldn't use the covariance matrix method anyway.

2. Barequet & Har-Peled mention the iterative refinement of OBB's using the caliper-fit. This is something I've known a while, but I've never seen it published before; I think it's one of those gems of wisdom that lots of people know but don't consider worth a paper. They mention it almost in passing, but it's actually perhaps the most valuable thing in their whole paper.

Recall if you have one axis of the OBB fixed, you can easily find the optimal directions of the other two axes using rotating calipers to fit a rectangle. The thing is, once you do that, you can then hold one of those new axes fixed, and fit the other two. So like, fix X, then caliper to get YZ, then fix Y, and caliper to get XZ. Each step of the iteration either improves your OBB or does nothing. That means you descend to a local minimum in a finite number of steps. (in practice I find you usually get there in only 3 steps, in fact that might be provable (?)).

Assuming your original seed box is pretty close to optimal, this iteration is kind of like taking your OBB and trying to spin it along one axis and pinch it to see if you get a tighter fit; it's sort of like wiggling your key as you put it into a lock. If your seed OBB is close to being right, but isn't supported by one of the necessary support conditions, this will wiggle it tighter until it is supported by a face or an edge pair.

The methods shown in the test below are :

True convex hull (within epsilon) :
    note that area optimization is usually what you want
    but volume shows a bigger difference between methods

Hull simplificiation :

    I simplify the hull by just doing PM on the triangles
    Then convert the triangles to planes
    Push the planes out until all points are behind them
    Then clip the planes against each other to generate new faces
    This is a simpler hull that strictly contains the original hull

k-dop :
    Fits 258 planes in fixed directions on the sphere
    Just pushes each plane to the edge of the point set
    Clips them all against each other
    strictly this is O(n) but in practice it's slower than the true convex hull
        and much worse quality
    seems pointless to me

The rating we show on all the OBB's is surface area

AxialOBB  :
    axis-aligned box

OBBByCovariance :
    vertex covariance matrix sets axes

OBBByCovariance+it :
    OBBByCovariance followed by iterative greedy optimization

OBBByCovarianceOptimized :
    like OBBByCovariance+it, but tries all 3 initial fixed axes

OptimalOBBFixedDirections :
    tries 42 fixed directions

OptimalOBBFixedDirections+it :
    tries 42 fixed directions, takes the best, then optimizes

OptimalOBBFixedDirections opt :
    tries 42 fixed directions, optimizes each one, then takes the best

OBBGoodHeuristic :
    takes the best of OBBByCovarianceOptimized and "OptimalOBBFixedDirections opt"

    this is OBBByCovarianceOptimized but using the solid body covariance instead of points  

OptimalOBB :
    tries all face normals of the convex hull (slow)
    I'd like to also try all the edge-support directions here, but haven't figured it out

OptimalOBBBarequetHarPeled 5     
 kLimit : 5 , numBuilds : 19
    BarequetHarPeled method with (i+j+k) <= 5
    causes it to try 19 boxes
    optimizes each one, then picks the best
    very similar to "OptimalOBBFixedDirections opt"

And the results :


dolphin.x :

    Made Hull with 206 faces
    hull1 volume : 1488557 , area : 95330

Making hull from k-dop planes...
    Made Hull with 142 faces
    hull2 volume : 2081732 , area : 104951

Making OBB...
    AxialOBB                         : 193363.109
    OBBByCovariance                  : 190429.594
    OBBByCovariance+it               : 179504.734
    OBBByCovarianceOptimized         : 179504.719
    OptimalOBBFixedDirections        : 181693.297
    OptimalOBBFixedDirections+it     : 181693.297
    OptimalOBBFixedDirections opt    : 176911.750
    OBBGoodHeuristic                 : 179504.719
    OBBGivenCOV                      : 178061.406
    OptimalOBB                       : 176253.359

     kLimit : 3 , numBuilds : 3
    OptimalOBBBarequetHarPeled 3     : 179504.703
     kLimit : 5 , numBuilds : 19
    OptimalOBBBarequetHarPeled 5     : 178266.047
     kLimit : 10 , numBuilds : 160
    OptimalOBBBarequetHarPeled 10    : 176508.109
     kLimit : 20 , numBuilds : 1222
    OptimalOBBBarequetHarPeled 20    : 176218.344
     kLimit : 50 , numBuilds : 18037
    OptimalOBBBarequetHarPeled 50    : 176116.156


teapot.x :

hull1 faces : 612
    hull1 volume : 3284935 , area : 117470

simplified hull2 faces : 366
    hull2 volume : 3384222 , area : 120357

Making hull from k-dop planes...
    Made Hull with 234 faces
    hull2 volume : 3761104 , area : 129271

Making OBB...
    AxialOBB                         : 253079.797
    OBBByCovariance                  : 264091.344
    OBBByCovariance+it               : 222514.219
    OBBByCovarianceOptimized         : 220723.844
    OptimalOBBFixedDirections        : 219071.703
    OptimalOBBFixedDirections+it     : 218968.844
    OBBGoodHeuristic                 : 218968.844
    OptimalOBB                       : 218968.844
    OBBGivenCOV                      : 220762.766

     kLimit : 3 , numBuilds : 3
    OptimalOBBBarequetHarPeled 3     : 220464.766
     kLimit : 5 , numBuilds : 19
    OptimalOBBBarequetHarPeled 5     : 219540.203
     kLimit : 10 , numBuilds : 160
    OptimalOBBBarequetHarPeled 10    : 218968.000
     kLimit : 20 , numBuilds : 1222
    OptimalOBBBarequetHarPeled 20    : 218965.406
     kLimit : 50 , numBuilds : 18037
    OptimalOBBBarequetHarPeled 50    : 218963.109


Some highlights :

OBBByCovariance is quite bad. OptimalOBBFixedDirections is the only other one that doesn't do the iterative optimization, and it can be bad too, though not nearly as bad.

Any of the methods that do the iterative optimization is perfectly fine. The differences are very small.

"OptimalOBBBarequetHarPeled 7" does about the same number of tests as "OptimalOBBFixedDirections opt" , and it's very microscopically better because of the way the directions are distributed.

OBBGivenCOV (the solid mass covariance) is worse than OBBByCovarianceOptimized (point covariance) on teapot.

Also - the Convex Hull simplification thing I did was just pulled out of my ass. I did a quick Google to see if I could find any reference, and I couldn't find any. I'm surprised that's not a solved problem, it seems like something right up the Geometers' alley.

Problem : Find the convex bounding volume made of N faces (or N planes) that strictly encloses the original mesh, and has minimum surface area (or volume).

In general, the optimal N-hull can not be reached by greedy simplification from the full-detail convex hull. In practice I found my hacky PM solution to work fine for moderate simplification levels. To make it more correct, the "push out to enclose" step should be done in each PM collapse to keep the hull valid as you go (instead of at the end). Also the PM collapse metric should be the metric you are trying to optimize - surface area or volume (I just used my old geometric error collapser).

The main thing I was interested in with convex hull simplification was eating away highly tesselated bits. The mesh I mainly tested on was "fandisk" because it's got these big flat surfaces, and then some rounded bits. If you imagine a mesh like a big cube minowski summed with a small sphere, you get a cube with rounded edges and corners. If the sphere is highly tessleated, you can get a hull with tons and tons of faces, but they are very unimportant faces. You want to sort of polygonate those corners, replaces the rounded sphere with a less tesselated one that's pushed out.

04-23-09 | iPod Hardware - the harbinger of suck

Is god fucking awful. Stop touting it as the greatest example of product design in this century. Yes, yes, the screen is nice, and the basic shape and weight of it is appealing. If it was just a paperweight I would be pretty pleased with it. But when you actually try to *use* it, it's rubbish. (it's the Angelina Jolie of product design if you will - it's the canonical example that everyone uses of something that's great, but it's actually awful).

Try to actually browse through the menus with that fucking wheel. Scan through a big list of artists, back up, change to album view, scan down, it's awful.

The wheel is a disaster for volume control. The right thing for volume is a knob, or a dial. Something physical that you rotate. And it shouldn't be a fucking digital dial that just spins like all the shit that they're giving us on computers now. It should be an *absolute* dial with an actual zero point, so that I can turn the volume down to zero when the thing is off. Hitting play on an iPod is fucking ear drum roulette, you never know when it's going to explode your head.

You're playing a song, you pause it, you go browse to some other song. You want to just resume the original song. How do you even do that !? I suppose it must be possible, but I don't know. It should just be a button.

Design for music playing devices has been perfect for a long time. You have play, pause, skip, volume. You have those on different buttons that are ALWAYS those buttons. They're physical buttons you can touch, so you can use the device while it's in your pocket or your eyes are closed. They should be rubber (or rubberized) so they're tactile, and each button should have a different shape so you know you're on the right one.

It's just fucking rubbish. It's like so many cars these days, changing user interfaces for no good reason and making them worse. Don't give me fucking digital buttons to increment and decrement the air conditioning, that's awful! Give me a damn dial or a slider that has an absolute scale. I don't want to be hitting plus-plus-plus.

The damn "Start" button that's on so many cars now really pisses me off. You used to stick in a key, then turn it. What's wrong with that? It works perfectly fucking fine. Now I have to stick in a key, make sure I'm pressing the brake or the clutch or whatever, then press a start button. Why !? It's more steps, it's just worse.

The worst of course is the menu shit like iDrive that's basically an iPod style interface with a fucking wheel and menus and context-dependent actions. Context-dependent actions are fucking horrible user interface design, quit it. With consumer electronic devices there should just be a few buttons, and those buttons always execute the same action and do it immediately. I know some jack-hole is going to run into me because he was trying to mate his bluetooth and was browsing around the menus.

04-23-09 | The Prurient Underbelly

Our cities are full of "massage parlors" that offer prostitution with a blatant store front and ads in the paper.

I wrote before about how every girl in LA does porn .

There are some widely spread numbers about $10 billion in porn movies per year, or 800 million rentals per year, or the number of porn movies that hotel guests rent. Unfortunatley those numbers are just made up ; (it's funny that even in his mea culpa about making up numbers he goes on to just make up other numbers out of thin air. good reporting, bub).

I was thinking about this because a few Sundays ago there was an article in the NYT Magazine about "SeekingArrangement.com". I didn't think much of it at first. SeekingArrangement is an online dating site like match or whatever, but it's specifically design for hooking up rich older men (often married) with young girls. It does toe the line of prostitution because it explicitly talks about dollar rates and what the girls would be willing to do. Still, these kind of arrangements have been around forever, I do find it rather disgusting, but it's not surprising.

But the sheer numbers are a bit surprising. They claim 300,000 girls have posted profiles. I assume they are exaggerating somewhat and some of the accounts are fake. Let's do some fake numbers like last time to get a surprising conclusion.

I assume most of the girls registered are in the age range 20-30. There are around 20M people in the US in the age range 20-30 (this number is correct BTW). About half of those are girls, so around 10 M. If the 300k girls on SeekingArrangement are from the US, that's 3% of the female population (!).

A more accurate number : there are around 200,000 girls in WA state in the target age group. There are around 2000 girls on SeekingArrangement from WA state. That's 1%. Still a rather high number.

Not really related, but also :

1. STOP TATTOOING YOURSELF !! My god, a whole generation of girls is ruining themselves. Fortunately I found a hot girl with no disgusting dark-green graffiti that breaks up the natural lovely smooth lines of the female body, but I still have to look at it on models and such. WTF. Oh yes, I know what would go great with this peach skin, this graceful natural arc of flesh and muscle, this magical body that is firm and yet soft, this curve that speaks right to my gut - a fucking dark green dolphin drawn by some high school dropout. Quit it.

2. Angelina Jolie is fucking revolting. She looks so bizarre, she's had so much plastic surgery, she looks like an alien. I'm so sick of her being used as the modern standard of beauty. She's the Garbo or Marilyn Monroe or Sofia Loren or whoever of today, and that just says volumes about the fucking awful taste of modern man. I hate that so many girls are trying to look like her and get that fucking punched-in-the-mouth lip injection I-cant-even-talk-right-cuz-my-lips-are-so-swollen.

04-23-09 | Telling Time

.. telling time is a huge disaster on windows.

To start see Jon Watte's old summary that's still good .

Basically you have timeGetTime() , QPC, or TSC.

TSC is fast (~ 100 clocks) and high precision. The problems I know of with TSC :

TSC either tracks CPU clocks, or time passing. On older CPUs it actually increments with each cpu cycle, but on newer CPUs it just tracks time (!). The newer "constant rate" TSC on Intel chips runs at some frequency which so far as I can tell you can't query.

If TSC tracks CPU cycles, it will slow down when the CPU speedsteps. If the CPU goes into a full sleep state, the TSC may stop running entirely. These issues are bad on single core, but they're even worse on multi-proc systems where the cores can independently sleep or speedstep. See for example these linux notes or tsc.txt .

Unfortunately, if TSC is constant rate and tracking real time, then it no longer tracks cpu cycles, which is actually what you want for measuring performance (you should always report speeds of micro things in # of clocks, not in time).

Furthermore on some multicore systems, the TSC gets out of sync between cores (even without speedsteps or power downs). If you're trying to use it as a global time, that will hose you. On some systems, it is kept in sync by the hardware, and on some you can get a software patch that makes rdtsc do a kernel interrupt kind of thing which forces the TSC's of the cores to sync.

See this email I wrote about this issue :

Apparently AMD is trying to keep it hush hush that they fucked up and had to release a hotfix. I can't find any admission of it on their web site any more ;

this is the direct download of their old utility that forces the cores to TSC sync : TscSync

they now secretly put this in the "Dual Core Optimizer" : Dual Core Optimizer Oh, really AMD? it's not a bug fix, it's an "optimizer". Okay.

There's also a seperate issue with AMD C&Q (Cool & Quiet) if you have multiple cores/processors that decide to clock up & down. I believe the main fix for that now is just that they are forbidden from selecting different clocks. There's an MS hot fix related to that : MS hotfix 896256

I also believe that the newest version of the "AMD Processor Driver" has the same fixes related to C&Q on multi-core systems : AMD Driver I'm not sure if you need both the AMD "optimizer" and processor driver, or if one is a subset of the other.

Okay, okay, so you decide TSC is too much trouble, you're just going to use QPC, which is what MS tells you to do anyway. You're fine, right?

Nope. First of all, on many systems QPC actually is TSC. Apparently Windows evaluates your system at boot and decides how to implement QPC, and sometimes it picks TSC. If it does that, then QPC is fucked in all the ways that TSC is fucked.

So to fix that you can apply this : MS hotfix 895980 . Basically this just puts /USEPMTIMER in boot.ini which forces QPC to use the PCI clock instead of TSC.

But that's not all. Some old systems had a bug in the PCI clock that would cause it to jump by a big amount once in a while.

Because of that, it's best to advance the clock by taking the delta from previous and clamping that delta to be in valid range. Something like this :

U64 GetAbsoluteQPC()
    static U64 s_lastQPC = GetQPC();
    static U64 s_lastAbsolute = 0;

    U64 curQPC = GetQPC();

    U64 delta = curQPC - s_lastQPC;

    s_lastQPC = curQPC;

    if ( delta < HUGE_NUMBER )
        s_lastAbsolute += delta;

    return s_lastAbsolute;

(note that "delta" is unsigned, so when QPC jumps backwards, it will show up as as very large positive delta, which is why we compare vs HUGE_NUMBER ; if you're using QPC just to get frame times in a game, then a reasonable thing is to just get the raw delta from the last frame, and if it's way out of reasonable bounds, just force it to be 1/60 or something).


BTW while I'm at I think I'll evangelize a "best practice" I have recently adopted. Both QPC and TSC have problems with wrapping. They're in unsigned integers and as your game runs you can hit the end and wrap around. Now, 64 bits is a lot. Even if your TSC frequency is 1000 GigaHz (1 THz), you won't overflow 64 bits for 194 days. The problem is they don't start at 0. (

Unsigned int wrapping works perfectly when you do subtracts and keep them in unsigned ints. That is :

in 8 bits :

U8 start = 250;
U8 end = 3;

U8 delta = end - start;
delta = 8;

That's cool, but lots of other things don't work with wrapping :

U64 tsc1 = rdtsc();

... some stuff ...

U64 tsc2 = rdtsc();

U64 avg = ( tsc1 + tsc2 ) /2;

This is broken because tsc may have wrapped.

The one that usually gets me is simple compares :

if ( time1 < time2 )
    // ... event1 was earlier

are broken when time can wrap. In fact with unsigned times that wrap there is no way to tell which one came first (though you could if you put a limit on the maximum time delta that you consider valid - eg. any place that you compare times, you assume they are within 100 days of each other).

But this is easily fixed. Instead of letting people call rdtsc raw, you bias it :

uint64  Timer::GetAbsoluteTSC()
    static uint64 s_first = rdtsc();
    uint64 cur = rdtsc();
    return (cur - s_first);

this gives you a TSC that starts at 0 and won't wrap for a few years. This lets you just do normal compares everywhere to know what came before what. (I used the TSC as an example here, but you mainly want QPC to be the time you're passing around).

04-22-09 | Spam

WTF gmail, why do you keep letting obvious nonsense through?

I literally get around 10 mails like this a day :

From: Susan Brown <susannjtnpocs@hotmail.com> 

Subject: Authentic discounted prescriptions from Canada. 

Fast delivery and qualitative support- come and see yourself.


How can you not tell that's spam !?

On the plus side, today it delivered one of the more amusing spams I've ever gotten :

I want you to read this message very carefully, and keep the secret with you till further notice, You have no need of knowing who I am, where am from, till I make out a space for us to see, I have being paid me $15000 in advance to terminate you with some reasons listed to me by my employer, it's one I believe you call a friend, I have followed you closely for one week and three days now and have seen that you are innocent of the accusation.

Do not contact the police or security agent or try to send a copy of this to them, because if you do I will know, and might be pushed to do what I have being paid to do,,beside, this is the first time I turned out to be a betrayer in my job. Now, listen, I will arrange for us to see face to face but before that you have to pay me some $20,000. I will be coming to see you in your office or home determine where you wish we meet, do not set any camera to cover us or set up any tape to record our conversation, my employer is in my control now, I will give you the full information of my employer and video and audio tape conversation with him that contains his request for me to terminate you, which will be enough evidence for you to take him to court (if you wish to).

You don’t need my phone contact for now till am assured you are ready to comply positively. Once more I shall remain anonymous, however, you have no alternatives other than to co-operate positively and do as I said or face the brewing storm. You must also trust that as long as you keep your part of the deal, I shall too keep mine and live up to my words. This time I am holding the trump card and shall put it to effective use if you fail to co-operate positively as requested. Any move by you other than co-operation will have catastrophic consequences to you. I wait for your response urgently to settle this and counter my employer directives against you immediately.

Thanks you for your attention Cross

I love the writing style. It's got that high-school-kid-trying-to-sound smart ring to it. "do as I said or face the brewing storm" - how precious! It reminds me a bit of VCB. "They sat in the cafe and discussed art and romance." Oh yes. I am such a passionate latin lover, I can't be bothered to button my shirt properly, and I love to listen to Spanish guitar and discuss romance.

04-17-09 | Bleck

We've seen some really awful movies recently.

"The Queen" : wow, WTF is the point of this movie. It's hard to imagine anything less interesting than the royal family, and despite what all the critics say, the portrait of them was completely stereotypical and superficial. In no way was it interesting or realistic or surprising. Plus like 50% of the time is spent on stock footage of the whole Diana death bullshit.

"Vicky Christina Barcelona" : another wowie zowie this was shockingly bad. People talk about this movie and "Match Point" as the reemergence of Woody Allen after a period of making predictable snoozers. Yes his 90's movies were pretty weak, but these are even worse. VCB is literally laugh-out-loud awful. I'm actually a little perplexed by it. I'm not quite sure if the whole movie is supposed to be a "level". The voice over for example is intentionally bad, right? I mean that voice over can't be serious, it's super cheezy bad like a romance novel or a Penthouse Forum letter, I assume/hope that's on purpose. I guess that all the critics that liked this movie are just horny sexually repressed nerds that were aroused by all the hints of sexuality.

I guess I'll go back to watching endless old Top Gear episodes.

The new High Stakes Poker cast is disappointing too. Hachem, Laak and Esfandiari are some of the biggest cocks in poker. They're all cry-baby attention whores with no real game skill or interesting personality. Laak might be the worst; he's a huge nit and just generally sucks; he normally plays $10/20 NL live and just bum-hunts. Last episode he made one of the most retarded plays in poker - raising to "defend" a weak hand. It's a play that tilts me so bad, because dumb "pros" like Laak or Gabe Kaplan will recommend it, and it usually works so they think they did the right thing. Basically if you flop a hand that is decent and probably best, but is not strong, you should almost never be raising with that. Like say you flop a weak middle pair or something like that. You just call, you don't raise. (obviously there are exceptions, everything in poker is situational, there are no formulas or set "moves").

04-17-09 | Oodle File Page Cache

I'm trying to figure something out, maybe someone out there has a good idea.

This is about the Oodle File Page Cache that I mentioned previously. I'm not doing the fully general page cache thing yet (I might not ever because it's one of those things you have to buy into my philosophy which is what I'm trying to avoid).

Anyway, the issue is about how to prioritize pages for reclamation. Assume you're in a strictly limitted memory use scenario, you have a fixed size pool of say 50 pages or so.

Now obviously, pages that are actually current locked get memory. And the next highest priority is probably sequentially prefetched pages (the pages that immediately follow the currently locked pages) (assuming the file is flagged for sequential prefetching, which it would be by default).

But after that you have to decide how to use the remaining pages. The main spot where you run into a question is : I need to grab a new page to put data into, but all the pages are taken - which one do I drop and recycle? (Or, equivalently : I'm thinking about prefetching page X, the least important page currently in the pool is page Y - should I reclaim page Y to prefetch page X, or should I just wait and not do the prefetch right now).

The main ambiguity comes from "past" pages vs. "prefetched" pages (and there's also the issue of old prefetched pages that were never used).

A "past" page is one that the client has unlocked. It was paged in, the client locked it, did whatever, then unlocked it. There's one simple case, if the client tells me this file is strictly forward-scan streaming, then the page can be dropped immediately. If not, then the past page is kept around for some amount of time. (there's another sequence point when the file containing the page is closed - again optionally you can say "just drop everything when I close the file" or the pages can be kept around for a while after the close to make sure you were serious about closing it).

A "prefetched" page obviously can be prefetched by sequential scan in an open file. It could also be from a file in the prefetch-ahead file list that was generated by watching previous runs.

Prefetches pages create two issues : one is how far ahead do I prefetch. Basically you prefetch ahead until you run out of free pages, but when you have no free pages, the question is do I reclaim past pages to do new prefetches?

The other issue with prefetches is what do you do with prefetched pages that were never actually used. Like I prefetched some pages but then the client never locked them, so they are still sitting around - at what point do I reclaim those to do new prefetches?

To make it more clear here's a sort of example :

Client gives me a prefetch file list - {file A, file B, file C, file D}

Client opens file A and touches a bunch of pages.  So I pull in {A:0,A:1,A:2} (those are the page numbers in the file).

I also start prefetching file B and file C , then I run out of free pages, so I get {B:0,B:1,C:0}.

Client unlocks the pages in file A but doesn't close file A or tell me I can drop the pages.

Client now starts touching pages in file C.  I give him C:0 that I already prefetched.

Now I want to prefetch C:1 for sequential scan, I need to reclaim a page.

Do I reclaim a page from B (prefetched but not yet used) or a page from A (past pages) ?  Or not prefetch at all?

When client actually asks for C:1 to lock then I must reclaim something.

Should I now start prefetching {D:0} ?  I could drop a page from {A} to get it.

Anyway, this issue just seems like a big mess so I'm hoping someone has a clever idea about how to make it not so awful.

There's also very different paradigms for low-page-count vs high-page-count caches. On something like the PS2 or XBox 1 where you are super memory limitted, you might in fact run with only 4 pages or something tiny like that. In that case, I really want to make sure that I am using each page for the best purpose at all times. In that scenario, each time I need to reclaim a page, I should reevaluate all the priorities so they are fresh and make the best decision I can.

On something like Windows you might run with 1024 pages. (64k page * 1024 pages = 64 MB page cache). In that case I really don't want to be walking every single page to try to pick the best one all the time. I can't just use a heap or something, because page priorities are not static - they can change just based on time passing (if I put any time-based prioritization in the cache). Currently I'm using a sort of cascaded priority queue where I have different pools of priority groups, and I only reevaluate the current lowest priority group. But that's rather complicated.

04-17-09 | Waffling

I'm waffling about what car to get. I really *need* a new car because I've realized that the Qualude is just too small for me. I've been sitting hunched over all my life, and I'm trying to correct that and decompress my spine, and I just can't do it.

I'm waffling about whether to have surgery. Some days my shoulders feel sort of okay and I think "I could live with this" but then other days I'm in a lot of pain and it's just all crunchy and grinding and searing pain and awful. And of course any time I try to do a pushup I think "I need surgery".

I always see both sides of every decision, and it's quite paralyzing. In my opinion, most people, even smart people are very cavalier about ignoring the pros and cons. But they are probably right to do so. It's better to just pick something and go with and do your best with that decision. To put it in programming terms - I would say 90% of the smart programmer that I know are far too opinionated about various style issues, they don't rationally give fair credit to the other side's arguments and see that there are good arguments on both sides. But that's fine. Somebody who just says "low level C-style is the one true way" and runs with it might be wrong, but they can be productive because they have just made a decision and are getting work done. Somebody who sits around all the time thinking "hmm would it be better to do this C-style, or C++ OOP? or maybe need a GC language, or I could use OCaml for this bit.." is wasting way too much time equivocating.

Most successful business men, and just dynamic and fun people that I admire, are quick decision makers that just pick something and stick with it.

A related topic I've been thinking about recently : I've always been very dubious about the idea of learning from people who have been successful. There's this whole cult of worshipping rich people, reading interviews with them, getting their opinions on things, trying to learn what made them successful. I think it's mostly nonsense. The thing is, if you just look at who the biggest earners are, it's almost entirely luck.

Think about it this way - you have a bunch of gamblers. Some of them are very good and will make a steady profit of +10 a year with not much variance. The others suck, but play very wild, so their expected profit is +0, but they have high variance, so it could go between -100 and +100 in any given year.

If you look at the list of who the top winners are in any given year, it will be the people who suck. It will just be the ones that happened to get lucky. If you also look at the biggest losers, they will be using the exact same strategy as the biggest winners.

To make a more concrete example - lets say you have a biased coin you're allowed to flip. It has a 55% chance of being heads and you can bet as much of your bankroll as you want on it. A smart gambler might use a "Kelly" bet size ( see my old blog post ) and make a nice expected profit. A crazy gambler would just bet their whole bankroll on every flip. If you have 1000 people betting smart, and 1000 people betting crazy , after 10 flips all the biggest winners will be people playing the crazy strategy.

Anyway, the point is if you just look at successful business people, they will probably be confident, decisive, risk takers, aggressive at seizing opportunities, aggressive about growing the business quickly, etc. That doesn't mean that those are the right things to do. It just means that those are variance-increasing traits that give them a *chance* to be a big success.

Now, just like in poker, there are times when variance-increasing is the right move. For example, if you believe that you basically suck. If you think you don't have an edge on your opponent, then the best way to beat him is to just play really wild and hope you get lucky. For example, the best way for a novice poker player to have a chance in a tournament is to just do lots of all-in preflop shoving and hope you win the draws. The same thing could certainly be true in capitalism; if you believe you don't really have any great money-making skills, your best way to get rich is to variance it up.

04-16-09 | The Daily Ramble

Seattle is getting fucked. They're destroying Capital Hill. It's still sort of charming now, with old brick buildings, lots of greenery and plants, cheap rents and hipsters and artists. But they are tearing down the cool old buildings as fast as they can and putting up awful generic big blocks of condos plus retail space. They are literally turning it into Bellevue as fast as they can.

I despise dealing with the fucking receptionists at all these doctor places. I'm pretty sure the receptionist at MTI Physical Therapy is an alcoholic. I feel like I can identify it with just a minute of contact. There's a certain lazy attitude, a sloppy way of sitting, and a sort of floppy facial muscle nature, the way they talk is like their lips are too big and loose and they have to chew out the words. She's also got the sort of big jaw and brow and large nose with the bad pitted skin that I believe is caused by liver damage (Petechiae).

Anyway, it makes me think it might be nice to have a personal assistant. In theory you don't actually need to be making all that much money for a PA to be worth it. If they could actually remove all these distractions for me and help me focus on work and have a clear mind, it would be a huge boon. I don't want to have to think about bills or car registration or talking to receptionists or shopping or anything. I want only work and pleasure. It would give me time to exercise more, and to do more hobby programming. My concern would be that I couldn't actually trust the PA. Not "trust" as in worry they would steal from me, but trust as in be confident that they are taking care of things correctly. If I can't trust them, then it doesn't free my mind from worrying about all those things; I have to have full confidence that they are on top of everything, and it seems like someone solid enough to take care of things is not going to be a PA.

The tulips have started opening up around the city. I love the way the spring bloom here has such discrete stages. In a lot of places (Texas) it just suddenly happens one day and everything is blooming, but here we've had these steps as different species hit their time one by one. It's also pretty amazing the way the bulbs around here naturalize and just pop up in random parking strips and between bricks and such; we've had crocus, daffodil, and now the tulips.

I found a nice way to get a bike up and down cap hill - Interlaken Drive in Interlaken Park. It's like a tiny bit of woods, and there's nary a car on it. It's also got the really cool Hebrew Academy on it, which looks like some old gothic mansion; it makes me think of Wuthering Heights or Dickens or something, I imagine a rich old man muddling around in its dusty halls with a terribly unhappy neice that he forbids to go outside. I put some pictures on my flickr .

04-15-09 | Oodle Page Cache

So I'm redoing the low level file IO part of Oodle. Actually I may be retargetting a lot of Oodle. One thing I took from GDC and something I've been thinking about a long time is how to make Oodle simpler. I don't want to be making a big structure that you have to buy into and build your game on. Rather I want to make something "leafy" that's easy for people to plug in at the last minute to solve specific problems.

Pursuant to that, the new idea is to make Oodle a handful of related pieces. You can use one or more of the pieces, and each is easy to plug in at the last minute.

1. Async File IO ; the new idea with this is that it's cross platform, all nice and async, does the LZ decompression on a thread, can do DVD packaging and DVD emulation, can handle the PS3/Xenon console data transfers - but it just looks like regular files to you. This is less ambitious than the old system ; it no longer directly provides things like paging data in & out, or hot-loading artist changes; you could of course still do those things but it leaves it more up to the client to do that.

The idea is that if you just write your game on the PC using lazy loose file loads, boom you pop in Oodle and hardly touch the code at all, and you automatically get your files packed up nice and tight into like an XBLA downloadable pack, or a DVD for PS3, or whatever, and it's all fast and good. Oh, and it also integrates nicely with Bink and Miles and Granny so that you can things like play a Bink video while loading a level, and the data streamers share the bandwidth and schedule seeks correctly.

2. Texture goodies. We'll provide the most awesome threaded JPEG decoders, and also probably a better custom lossy and custom lossless texture compressors that are specifically designed for modern games (with features like good alpha-channel support, support for various bit depths and strange formats like X16Y16 , etc.). Maybe some nice DXTC realtime encoding stuff and quality offline encoding stuff. Maybe also a whole custom texture cache thing, so you can say Oodle use 32 MB for textures and do all the paging and decompression and such.

3. Threading / Async utilities. You get the threaded work manager, the thread profiler, we'll probably do "the most awesome" multithreaded allocator. We'll try to give these specific functions that address something specific that people will need to ship a game. eg. a customer is near ship and their allocator is too slow and taking too much memory so they don't fit in the 256 MB of the console. Boom plug in Oodle and you can ship your game.

Anyway, that's just the idea, it remains to be worked out a bit. One thing I'm definitely doing is the low level IO is now going through a page cache.

As I'm writing it I've been realizing that the page cache is a super awesome paradigm for games in general these days. Basically the page cache is just like an OS virtual memory manager. There's a certain limited amount of contiguous physical memory. You divide it into pages, and dynamically assign the pages to the content that's wanted at the time.

Now, the page cache can be used just for file IO, and then it's a lot like memory mapped files. The client can use the stdio look-alike interface, and if they do that, then the page cache just automatically does the "right thing", prefetching ahead pages as they sequentially read through a file, etc.

But since we're doing this all custom and we're in a video game environment where people are willing to get more manual and lower to the bone, we can do a lot more. For example, you can tell me whether a file should sequential prefetch or not. You can manually prefetch at specific spots in the file that you expect to jump to. You can prefetch whole other files that you haven't opened yet. And perhaps most importantly, you can assign priorities to the various pages, so that when you are in a low memory situation (as you always are in games), the pages will be used for the most important thing. For example you can prefetch the whole next file that you expect to need, but you would do that at very low priority so it only uses pages if they aren't needed for anything more urgent.

The next awesome thing I realized about the page cache is that - hey, that can just be the base for the whole game memory allocator. So maybe you give 32 MB to page cache. That can be used for file IO, or video playback - or maybe you want to use it to pop up your in game "pause menu" GUI. Or say you want to stream in a compressed file - you map pages to read in the packed bits, and then you map pages as you decompress; as you decompress you toss the pages with the packed bits and leave the uncompressed pages in the cache.

Say you have some threaded JPEG decompress or something - it grabs a page to decompress to. The other cool thing is because all this is manual - it can grab the page with a certain priority level. If no page is available, it can do various things depending on game knowledge to decide how to respond to that. For example if it's just decompressing a high res version of a texture you already have in low res, it can just abort the decompress. If it's a low priority prefetch, it can just go to sleep and wait for a page to become available. If it's high priority, like you need this texture right now, that can cause other pages that are less important to get dropped.

Pages could also cache procedurally generated textures, data from a network, optional sounds, etc. etc.

I think about it this way - in the future of multicore and async and threading and whatnot, you might have 100 jobs to run per frame. Some of those jobs need large temp work memory. You can't just statically assign memory to various purposes, you want it to be used by the jobs that need it in different ways.

04-15-09 | PEPPR

There's a black Porsche Cayenne in Seattle with the custom plate "PEPPR". That on its own is reason enough to punch this guy in the crotch, but he's also a complete douchebag driver. He was speeding up and slowing down, tailgating, cutting across many lanes of traffic. He cut way across the merge triangle from 520 to 405, which kicks up a ton of debris, and he also did it straight into heavy traffic, causing lots of people to have to brake for him. What a cock. If you ever see this guy, please run into him intentionally, or at least key his car.

It is your moral duty as someone alive to punish dicks. We are all investors in the "social economy" that we all play in. If you don't punish dicks, you're just as bad as the pension managers who were asleep at the wheel of corporate oversight.

BTW you may be thinking it's hypocritical of me to criticize crazy drivers. Yes, I am somewhat crazy myself, but I try to be crazy without inconveniencing others. I try to be considerate. I hate people who drive too slow, but I also hate people who drive too crazy. That's not a contradiction - the point is that like most things in life there is a good middle ground, and you are a dick if you go too far in either direction.

04-15-09 | Bodies and Computers

I'm trying desperately to avoid sitting at the computer when I'm not working, but so far I haven't been too successful. I just don't know what else to do with myself. Last weekend I tried to make myself do all kinds of non-computer stuff. I put up drape hold backs, I vacuumed the radiators, I tacked the cable TV line to the floorboards, I went shopping for clothes around the city, I went for walks. I made a pork shoulder ragu with papardelle (that was quite fantastic, maybe the best ever ; similar to this or this ; some mods I made : use a mix of stock and red wine for braising liquid; when the braise is done, remove the meat and boat-motor the veg to make a sauce, use a bit of fennel seed, and one dried red chile).

And that all only took a few hours. Now what? I can read, watch TV, or get on the computer.

I kinda want to make a little iPhone game, but I'm not letting myself because that would mean a lot more computer sitting. (it would also surely distract me from the work I need to be doing for RAD).

If you are a young man thinking about a career, I strongly encourage you to stay away from computers. They will ruin your life. You will spent your years trapped in cubicles under flourescent lights, hanging out with a bunch of other nerdy men, not doing anything really dynamic or exciting in your life, and slowly ruining your body through keyboarding and too much sitting. You won't have contact with girls, or fun different people, you won't do things that you can tell stories about, you won't have freedom to develop your other interests, it will take all your time and you'll wind up getting pigeon-holed into some specialty that becomes increasingly tedious to you.

ADDENDUM : I don't think anyone sees my addendums now, because the first draft gets sent out with RSS, and then nobody goes back to the original page to see the updates. Mmm I kinda hate RSS. Anyhoo -

I got a medical massage from this guy named John Bagley at Belltown Healing Arts. It was fantastic, I felt a ton better afterward, and actually went in for a doctor exam later that same day, and my function tests were way improved. He's literally the first person I've seen that I felt like really examined my body and identified the problems. If you have computer/injury body problems, I highly recommend him, though it does mean you will have to be touched by a man, which may be confrousing to some.

If you go in to an orthopedist or a physical therapist and they don't tell you to take your shirt off, and actually look at and touch your muscles and bones to see what's going on - just walk out. They're a fucking hack. How can they diagnose you and treat you without seeing what's going on? Furthermore, a physical therapist should actually have a finger on your muscles as you do the exercises. If they just sit there and look bored and you do your reps, walk the fuck out.

04-13-09 | Fashion Chicken

I think fashion is kind of like a game of chicken. When I see someone that makes me stop and go "wow that's a cool look" it's almost always because they're wearing something really ridiculous, something that actually just looks goofy and retarded. But by wearing something ridiculous and pretending that it's okay, they are showing that they have enough status and coolness to get away with it. They're daring the world to laugh at them.

In PUA lingo, this is DHV (Demonstration of Higher Value). When someone wears something really outrageous it's like making a poker bluff saying "I am cool enough to get away with this outfit". They put that bet out there, and then the world can either call their bluff (by laughing at them), or if the world lets them get away with it, they have succeeded with the DHV.

Of course most people, like me, chicken out and don't even try to be outrageous. Wearing bland ordinary stuff is a way of being meek, invisible, of not trying, because you're afraid your bluff will fail and the world will laugh at you.

04-13-09 | Anger

My fucking physical therapist lied to me about being a preferred provider for my PPO, so now I get to go through health care grievance arbitration. Yay ! I'm not too optimistic about that working out well, so I probably get to eat the cost. Fuckers.

In other news our fucking cock landlord has the heat in the building turned off and it's been super cold again recently, so we had literally no heat over the weekend and it was miserable, so I got the fun of calling him.

Oh, and the white trash neighbors had a backyard bon fire at midnight and filled our house with smoke. That's the last one I'm tolerating, next time I just tell them no, and if it persists they get reported for fire code violations. I went and talked to them about it once before and they were just like "oh yeah okay". I have no problem confronting them, I'm just concerned about the possible revenge response. (* see later).

And the other side neighbors just had a baby. It's not actually that bad, we hear it cry a bit once in a while, but the side walls in the building are almost half decent (unlike the floors). The baby neighbors are actually incredibly considerate about not making noise, actually too considerate, I find it annoying. It makes me feel uptight just being around them. I sort of imagine that they put a pillow over the crying baby because they're worried about it being too loud.

The whole situation with health care billing pisses me off so much. You don't get to see your bill until weeks after you go in, because the health plan has to approve or not approve various charges. Many providers illegally try to "balance bill" you. Watch out for that. Even if you are careful and specifically try to go to a preferred providers, they will often take you in for xrays or something and the xray tech is not a preferred provider, so blammo suddenly you get a nasty bill. The thing that happened with my physical therapist is that the company is a preferred provider, but the particular therapist that they assigned me to is not.

It's ridiculous how bad the health insurance system is if you think about it. The health insurance companies should want you to go to cheaper, better providers. It would save them money, which should be the whole point of an HMO type company, is to set you up with better care for cheaper. But they don't, they do nothing. The right way to do it would be to provide information and choise to the customer. For example, make you pay a 10% coinsurance, and then show you the average cost of various providers. Also, they should give you quality ratings for various providers. It is in their best interest financially to send you to better quality providers who make you healthier in the minimum number of visits, because that reduces their cost of followups and complications. They are better off if you see surgeons who have a higher improvement rate. But they provide you with none of that information. Instead they literally just give you a provider directory with 1000 names in it. Gee thanks, now what do I do? Eenie meanie miney moe.

Some random less bitter junk :

This is pretty old, but the NYT Buy vs. Rent Calculator is super awesome. It's just an amazingly well designed piece of interactive statistics graphing. You can click on all the spots to get more info, you can drag the sliders on the left and it adjusts the graph. It's just so money.

The crashing real estate market is sort of tempting me, but this actually reminded me that it's not so hot. Yes, things are closer to a decent bargain than they have been, but even undervalued stuff is not likely to go up much at all in the next 5 years, what with finances being generally tight and people being scared of buying. I'm expecting home prices to be almost level for 5 years, so even getting into something that seems like a bargain right now isn't really a great move unless you hold it for 10 years or so.

(*) : I think the fear of revenge response is actually usually wrong. For example in customer service situations, when someone is being totally incompetent, like the other day I went to the UPS store and was trying to send something to Lebanon, PA, and the guy starts asking about declared value and contents and whatnot, I started thinking in my head "wait, are you doing a custom form to send this to the country of Lebanon?" but I just tried to be nice. The logic is that if you just call them on their stupidness then they will be unhelpful, while if you are polite and nice to them, they will try to help you more and give you better service.

I think that's mostly wrong. Most people just try to incompetently muddle through life and are just constantly fucking up and cutting corners and being lazy and not doing their job. If you let them, they will keep doing that. If you give an inch by being nice, they will just keep fucking up. If you call them on and say "hey, I'm sending this to Pennsylvania buddy", they get shocked out of it, they realize they can't get away with sneaking their half assed shit past you, because you're going to call them on it. Then they want to get their work done and take care of you because they don't want to be around someone that calls them on shit.

Also : Majesco's stock ticker is COOL !? WTF that is so awesome. And it's perfect if you ever meet the Majesco people. They know absolutely nothing about video game development, and literally talk like The Sopranos; I swear that company is some kind of mob money laundering operation.

04-08-09 | Link

Right now in Seattle, the Lawrimore Project features "Stability" - two people are living inside a giant teeter totter, so that they have to balance each other all the time.

Bakery Nouveau in West Seattle is probably the best bakery I've found yet here. It seems they only do one early morning baking, so late-day stuff is a bit stale. It's worth going in the morning on the weekend, fantastic pastries, bread, quiche and etc.

Skagit Valley is home to lots of Tulip and Daffodil farms where the fields are full of flowers for commercial production. It's quite a pretty area. The main bloom is coming in the next few weeks . I might try to get up there for a little road trip.

D & M seems to be one of the few places that will ship liquor to WA. Nice scotch selection.

Blackbird in Ballard seems to be the only actual cool men's clothing store in Seattle. Rather pricey.

You may know I've been looking at getting a 135 (damn Ryan beat me to it). One interesting option is Performance Center Delivery . You get the car cheaper, and you get a free track day out of it, which would be awesome. The down side is it's in fucking South Carolina (what kind of moron would ever want to live in SC!) which is an awfully far way to drive back here.

CoreInfo is useful for seeing how your cores map to logical processors and all that nonsense which will become ever more critical. I find all the low level lock free multithreading stuff super amusing to program, but it's just really not practical most of the time, because it does make code much harder to modify and debug and everything.

Kubota Garden is a large Japanese Garden in Seattle that was build almost entirely by a single family over many years. The city now runs it. Will have to check that out.

Good links from other blogs : Meme timeline and radical cartography .

Gus Hansen TV is pretty fucking awesome. He's quite a nutter. I think the jury is actually still out on whether he is any good or not. Most of the top players seem to think he is a huge fish - the high stakes games actually run *around* Gus right now, but Gus is not actually losing money in them yet. Time will tell. It requires Silverlight and a decently fast connection.

Some Puget Sound area touring links. I haven't done any of this yet so I don't know what of this information is actually good :

Trails and trips for Washington hiking, free info and resources...
SummitPost - Mountain Loop Highway -- Climbing, Hiking & Mountaineering
State Scenic Byway Scenic Driving Tours Washington State
National Scenic Byways Washington
Mountain Loop Highway, Glacier Peak Region, Washington
GORP - Washington Scenic Drives
Beat back the forest to exquisite Bedal Basin
Alpinism in the Pacific Northwest

04-08-09 | Knife Dipping

When you're putting condiments on your sandwich, you have to go in the right order. If you want mustard, mayo, and butter (btw butter on french bread is what makes a sandwich yum) you should go butter first, then mayo, then mustard. Getting a little butter in the mayo is no big deal, same with getting a little mayo in the mustard, but getting mustard on the butter would be a major faux-pas.

It's a trickier question with peanut butter and jelly. Do I get the PB in the jelly, or the jelly in the PB? And BTW please don't suggest using two knives, that is right out.

I've been going to a lot of physical therapy and doctor appointments and whatnot recently. It's such a huge distraction. For one thing it just takes a ton of time. The appointment might be only an hour, but you have to get there & park, then wait because they're always off schedule, then drive back. It winds up taking about three hours. And then there's the fact that it breaks up your day. I can't concentrate for hours before an appointment because I know an appointment is coming up. (it's kind of like how if I know I have to wake up early for something, I can't sleep that whole night because I keep waking up every half hour and looking at the clock wondering if I'll sleep through the alarm somehow). The whole hour before the appointment I can't really do any work because I don't want to start digging into something and get on a roll if I'm gonna have to cut it off.

The result is that on these days with doctor shit, I hardly get a lick of work done, and I just can't get my mind back onto focus. I only really do awesome work when I can wake up and just start working and not have any appointments or anything scheduled. I need to just be able to work as long as I want, when I want.

It also gets really pricey. At close to $100/visit at 3 visits to different people a week, you get into $1000/month very fast. I guess poor people don't get to be healthy. One of the weird things is that even though the physical therapist visit charges $200, (and I pay $50-$100 out of pocket that the insurance doesn't cover), the actual therapist is not making much money at all, they get maybe $30/hour ($60k/year). $170 is going to overhead, to the owner of the PT business, to the health insurance company and its executives and stock holders.

This is basically the model of all commerce in America these days - the product is too expensive for median-income consumers, and yet the people who actually make the product don't get that money, and they aren't paid enough to buy the very thing they make. The difference is getting skimmed to the super-rich. This is how income inequality grows. As a consumer, you should demand higher quality goods for cheaper prices, which means more of your money is going directly to the people who actually made the goods or provided the services. You should refuse to buy expensive garbage where 50%+ of the cost goes to management and shareholders.

Wright Angle is another nice quality Seattle food blog. Not awesome humor content like Surly Gourmand, but lots of good info and photos. It's helpful to me just to have blog subscriptions related to the things I like to do in life, because it gives me little reminders to get out and do those things. I'd like to find one about cycling around Seattle, maybe one about day trips and driving tours in the Puget Sound area, maybe one about raves, one about S&M and swinging, you know, all my interests.

Drew sent me this : NVidia Ion mini PC . I love these little mini cheap quiet PC's. NVidia might save their company with the cheap integrated market segment, because their integrated controller/graphics part is by far the best on the market right now (my god AMD/ATI should have such a clear advantage in this segment but they just can't get their shit together).

Anyway, the Ion PC made me think about something Butcher's been saying that just finally really rang true to me : the real loser in these PC price wars is MS. I mean obviously that's not true, clearly the hardware guys are the first to take the big hit, like Dell is in trouble because the Micro PC's are driving down prices and profit. But once we get into $300 and cheaper PC's, the problem for MS is that a $50-$100 copy of Windows doesn't make much sense any more.

When a PC is $1000 you can easily hide the $100 price of Windows, it doesn't seem so significant. But if you look at a $200 mini PC running Linux vs. a $300 mini PC with Windows, people will go for the Linux. (MS is aware of this and has recently started selling Windows cheaper to OEM's for use in the mini-PC market).

04-07-09 | Unbelievable

Fucking doctors are such fucking cock ass mother fucking scum sucking sons of bitches.

I've requested my medical records from every doctor that's been involved with my shoulders. I've gotten them from zero. The offices always treat me like I'm being such a huge pain in the ass. Fuck you it's my fucking records. When I tell them that I just referred myself they give me all this attitude. Fuck you, I have a PPO and I have fucking torn shoulders, I know I should be seeing a shoulder specialist, I don't need to go to some fucking primary care retard just to get referred back to you.

The doctors spend less than 5 minutes actually looking at me. They don't ask about detailed history at all. They immediately send you for MRI's. But then they literally don't even look at the MRI's. They just read the one page summary by the MRI tech. I've read those summaries and they're literally like one or two sentences. The doc doesn't read it before you visit, they just skim the summary once they get handed your chart in the office. Then they refer you to Physical Therapy.

The PT and the doc literally never talk. In fact, the doc doesn't even write any notes or instructions for the PT. When you show up at the PT office, they don't have your medical records or know anything about you really. WTF am I paying all you fucking tards so much for !?

Even at the PT places, the actual trained PT only sees you for maybe half an hour or so, and then you get handed off to the exercise babysitter who doesn't know shit about shit.

04-06-09 | Washington State Liquor Laws

The Washington State Liquor Laws are literally a direct funnel of money to the big distributors and producers. I knew that intuitively, but I found a good article that lays out the details : here . The law literally prohibits retailers from negotiating prices with distributors, prevents retailers from buying directly from producers, and mandates a 10% markup. This kind of public-private monopoly is always bad for the public, but is quite popular with both Dems and Reps (see, eg. health care, power utilities, telecom, military contractors, etc).

04-06-09 | Seattle Spring

is gorgeous. But it is wreaking havoc with my sinuses. It's gotten so bad that I'm getting bloody noses now, which is quite rare for me. It's especially fun when you have a bloody nose combined with hard sneezes, so that you shoot red snot all over the place. Anyway, walking around in the sunshine is delightful :

The azaleas have just started blooming in the warmer/sunnier spots around the city. I guess they'll kick in over the next few weeks. After that I'm looking forward to the Rhododendrons; I see them all over the city with their big leaves and their closed up flower buds, so I suspect there will be lots of them when they open. I think the arboretum has a nice collection.

While the cherries are nice, I'm particularly attracted to the trees with big white flowers; I'm not sure what they are, maybe dogwood? It's not something I've ever seen in California.

04-06-09 | The Work Dispatcher

I thought I'd write a bit about my multithreaded "Worklet" dispatcher before I forget about it. I call little units of work "worklets" just because I like to be nonstandard and confusing.

The basic idea is that the main thread can at any time fire up a bunch of worklets. The worklets then go to a bunch of threads and get done. The main thread can then wait on the worklets.

There are a few things I do differently in the design from most people. The most common "thread pool" and "job swarm" things are very simple - jobs are just an isolated piece of independent work, and often once a bunch is fired you can only wait on them all being done. I think these are too limiting to be really generally useful, so I added a few things.

1. Worker threads that are not in use should go completely to sleep and take 0 cpu time. There should be a worker thread per processor. There might also be other threads on these processors, and we should play nice with arbitrary other programs running and taking some of the cpu time. Once a worker thread wakes up to do work, it should stay awake as long as possible and do all the work it can, that is, it shouldn't have to go to sleep and wait for more work to get fired so it can wake up again.

2. Worklets can have dependencies on other worklets. That way you can set up a dependency tree and fire it, and it will run in the right order. eg. if I want to run A, then B, then run C only after A & B are both done, you fire worklets {A}, {B} and {C: dependent on AB}. Dependencies can be evaluated by the worker threads. That's crucial because it means they don't need to stall and wait for the main thread to fire new work to them.

3. The main thread can block or check status on any Worklet. The main thread (or other game threads) might keep running along, and the worker threads may be doing the work, and we want to be able to see if they are done or not. In particular we don't want to just support the OpenMP style "parallel for" where we fork a ton of work and then immediately block the main thread on it - we want real asynchronous function calls. Often in games we'll want to make the main thread(s) higher priority than the worker threads, so that the worker threads only run in idle time.

The actual worker threads I implemented with a work-stealing scheme. It's not true work-stealing at all, because there is a concept of a "main thread" that runs the game and pushes work, and there's also the dependencies that need to be evaluated. All of the thread-thread communication is lock free. When the main thread adds new work items it just jams them onto queues. When the worker threads pop off work items they just pop them off queues. I do currently use a lock for dependency evaluation.

Traditional work stealing (and the main papers) are designed for operating system threads where the threads themselves are the ones making work. In that environment, the threads push work onto queues and then pop it off for themselves or steal from peers. There are custom special lock-free data structures designed for this kind of operation - they are fast to push & pop at one end, but also support popping at the other end (stealing) but more slowly. What I'm doing is not traditional work stealing. I have external threads (the "game threads") that do not participate in the work doing, but they can push work to the workers. In my world the workers currently can never make new work (that could be added if it's useful).

There are a lot of nice things about work stealing. One is you don't need a seperate dispatcher thread running all the time (which would hurt you with more context switches). Another is that workers who have cpu time can just keep jamming along by stealing work. They sort of do their own dispatching to themselves, so the threads that have the cpu do the work of the dispatching. It also offloads the dispatching work from the main thread. In my system, the workers do all work they can that's known to be depency-okay. Once that work is exhausted they reevaluate dependencies to see if more work can be done, so they do the dependency checking work for themselves off the main thread.

Another nice thing about work stealing is that it's self-balancing in the face of external activity. Anything running on a PC has to face lots of random CPU time being stolen. Even in a console environment you have to deal with the other threads taking variable amounts of time. For example if you have 6 cores, you want 6 workers threads. But you might also have 3 other threads, like a main thread, a gpu-feeding thread, and a sound thread. The main thread might usually take 90% of the cpu, so the worker on that core rarely gets any time, the gpu-feeder might usually take 50% of the cpu time on that thread, but in phases, like it takes 100% of the cpu for half the frame then goes idle for the other half. With work stealing your worker thread will automatically kick in and use that other time.

In order for the self balancing to work as well as possible you need small worklets. In fact, the possible wasted time is equal to the duration of the longest task. The time waste case happens like this :

Fire N work items , each taking time T to N cores
For some reason one of the cores is busy with something else so only (N-1) of them do work
Now you block on the work being done and have to wait for that busy core, so total time is 2T

Total time should have been N*T/(N-1)
For N large the waste approaches T

Basically the smaller T is (the duration of longest work) the more granular the stealing self-allocation is. Another easy way to see it is :

You are running N workers and a main thread
The main thread takes most of the time on core 0
You fire (N-1) very tiny work items and the (N-1) non-main cores pick them up
You fire a very large work item and core 0 worker picks it up

That's a disaster. The way to avoid it is to never fire single large work items - if you would have split that into lots of little work items it would have self-balanced, because the stealing nature means that only worker threads that have time take tasks.

For example, with something like a DXTC encoder, rather than split the image into something like N rectangles for N cores and fire off 1 work item to each core, you should go ahead and split it into lots of tiny blocks. Of course this requires that the per-work-item overhead is extremely low, which of course it is because we are all lock-free and goodness.

There are some things I haven't done yet that I might. One is to be a bit smarter about the initial work dispatching, try to assign to the CPU that will have the most idle time. If you actually did make nice tiny worklets all the time, that wouldn't be an issue, but that's not always possible. In the case that you do make large work items, you want those to go the cores that aren't being used by other threads in your game.

Another issue is the balance of throughput vs latency. That is, how fast does the system retire work, vs. how long does it take any individual work item to get through. Currently everything is optimized for throughput. Work is done in a roughly FIFO order, but with dependencies it's not gauranteed to be FIFO, and with the work stealing and variations in CPU Time assignment you can have individual work items that take a lot longer to get through the system than is strictly necessary. Usually this isn't a big deal, but sometimes you fire a Worklet and you need it to get done as quickly as possible. Or you might need to get done inside a certain deadline, such as before the end of the frame. For example you might fire a bunch of audio decompression, but set a deadline to ensure it's done before the audio buffers run out of decompressed data. Handling stuff like that in a forward-dispatched system is pretty easy, but in work-stealing it's not so obvious.

Another similar issue is when the main thread decides to block on a given work item. You want that item to get done as soon as possible by the thread that has the highest probability of getting a lot of CPU time. Again not easy with work-stealing since some worker thread may have that item in its queue but not be getting much CPU time for some reason.

04-06-09 | Multi-threaded Allocators

I've been reading a bit about Multi-threaded Allocators. A quick survery :

The canonical base allocator is Hoard . Hoard is a "slab allocator"; it takes big slabs from the OS and assigns each slab to a fixed-size block. In fact it's extremely similar to my "Fixed Restoring" allocator that's been in Galaxy3 forever and we shipped in Stranger. The basic ideas seem okay, though it appears to use locks when it could easily be lock-free for most ops.

One thing I really don't understand about Hoard is the heuristic for returning slabs to the shared pool. The obvious way to do a multi-threaded allocator is just to have a slab pool for each thread. The problem with that is that if each thread does 1 alloc of size 8, every thread gets a whole slab for itself and the overhead is proportional to the number of threads, which is a bad thing going into the massively multicore future. Hoard's heuristic is that it checks the amount of free space in a slab when you do a free. If the slab is "mostly" free by some measure, it gets returned to the main pool.

My problem with this is it seems to have some very bad cases that IMO are not entirely uncommon. For example a usage like this :

for( many )
    allocate 2 blocks
    free 1 block

will totally break Hoard. From what I can tell what hoard will do is allocate you a slab for your thread the first time you go in, then when you free() it will be very empty, so it will return the slab to the shared pool. Then after that when you allocate you will either get from the shared pool, or pull the slab back. Very messy.

There are two questions : 1. how greedy is new slab creation? That is, when a thread allocates an object of a given size for the first time, does it immediately get a brand new slab, or do you first give it objects from a shared slab and wait to see if it does more allocs before you give it its own slab. 2. how greedy is slab recycling? eg. when a thread frees some objects, when do you give the slab back to the shared pool. Do you wait for the slab to be totally empty, or do you do it right away.

The MS "Low Fragmentation Heap" is sort of oddly misnamed. The interesting bit about it is not really the "low fragmentation", it's that it has better multi-threaded performance. So far as I can tell, the LFH is 99.999% identical to Hoard. See MS PPT on the Low Fragmentation Heap

We did a little test and it appears that the MS LFH is faster than Hoard. My guess is there are two things going on : 1. MS actually uses lock-free linked lists for the free lists in each slab (Hoard uses locks), and 2. MS makes a slab list *per processor*. When a thread does an alloc it gets the heap for the processor it's on. They note that the current processor is only available to the kernel (KeGetCurrentProcessorNumber ). Hoard also makes P heaps for P processors (actually 2P), but they can't get current processor so they use the thread Id and hash it down to P which leads to occasional contention and collisions.

The other canonical allocator is tcmalloc . I still haven't been able to test tcmalloc because it doesn't build on VS2003. tcmalloc is a bit different than Hoard or LFH. Instead of slab lists, it uses traditional SGI style free lists. There are free lists for each thread, and they get "garbage collected" back to the shared pool.

One issue I would be concerned about with tcmalloc is false sharing. Because the free lists get shared back to a global pool and then redistributed back to threads, there is no real provision to prevent little items from going to different threads, which is bad. Hoard and LFH don't have this problem because they assign whole slabs to threads.

The Hoard papers make some rather ridiculous claims about avoiding false sharing, however. The fact is, if you are passing objects between threads, then no general purpose allocator can avoid false sharing. The huge question for the allocator is - if I free an object on thread 1 that was allocated on thread 2 , should I put it on the freelist for thread 1, or on the freelist for thread 2 ? One or the other will make false sharing bad, but you can't answer it unless you know the usage pattern. (BTW I think tcmalloc puts it on thread 1 - it uses the freelist of the thread that did the freeing - and Hoard puts it on thread 2 - the freelist of the thread that did the allocation; neither one is strictly better).

Both tcmalloc and the LFH have high memory overhead. See here for example . They do have better scalability of overhead to high thread counts, but the fact remains that they may hold a lot of memory that's not actually in use. That can be bad for consoles.

In fact for video games, what you want in an allocator is a lot of tweakability. You want to be able to tweak the amount of overhead it's allowed to have, you want to be able to tweak how fast it recycles pages for different block types or shares them across threads. If you're trying to ship and you can't fit in memory because your allocator has too much overhead, that's a disaster.

BTW false sharing and threaded allocators are clearly a place that a generational copying garbage collector would be nice. (this does not conflict with my contention that you can always be faster by doing very low level manual allocation - the problem is that you may have to give tons of hints to the allocator for it to do the right thing). With GC it can watch the usage and be smart. For example if a thread does a few allocs, they can come from a global shared pool. It does some more allocs of that type, then it gets its own slab and the previous allocs are moved to that slab. If those objects are passed by a FIFO to another thread, then they can be copied to a slab that's local to that thread. Nice.

It seems clear to me that the way to go is a slab allocator. Nice because it's good for cache-coherence and false sharing. The ops inside a slab can be totally thread-local and so no need to worry about multithread issues. Slabs are large enough that slab management is rare and most allocations only take the time do inside-slab work.

One big question for me is always how to get from an object to its owning slab (eg. how is free implemented). Obviously it's trivial if you're okay with adding 4 bytes to every allocation. The other obvious way is to have some kind of page table. Each slab is page-aligned, so you take the object pointer and truncate it down to slab alignment and look that up. If for example you have 32 bit pointers (actually 31), and you pages are 4k, then you need 2 to the 19 page table entries (512k). If a PTE is 4 bytes that's 2 MB of overhead. It's more reasonable if you use 64k pages, but that means more waste inside the page. It's also a problem for 64-bit pointers.

There are some other options that are a bit more hard core. One is to reserve a chunk of virtual address for slabs. Then whenever you see a free() you check if the pointer is in that range, and if so you know you can just round the pointer down to get the slab head. The problem is the reserving of virtual address.

Another option is to try to use pointer alignment. You could do something like make all large allocs be 4k aligned. Then all small allocs are *not* 4k aligned (this happens automatically because they come from inside a 4k page), and to get the base of the page you round down to the next lowest 4k.

04-06-09 | Continuing Vista Shit

The Vista lappy keeps having one little quirk after another. The latest one is that it would go to 100% CPU if left idle for a while. This was a little hard to track down, because as soon as you touch it to try to see what's using the CPU - it shuts off that task, so you can't catch it in the task manager.

So I knew it obviously had to be one of those "background optimization" services that Vista so kindly runs for you. So I looked in the Scheduled Tasks thing for anything that was set to run "when machine is idle".

It looks like the culprit was CrawlStartPages.

Eat "disable" motherfucker.

04-02-09 | Leather

Test drove a 135 again. It's a fucking great car, the steering feel and suspension are marvelous, and it also goes wonderfully tame and quiet if you lay off the throttle. But when the mood hits you, power is available at any moment because of the amazingly flat torque curve.

But the seats are leather. In about 5 minutes my ass was a sweltering swamp. My back was a waterfall draining into said swamp. Your only other seat option is plastic (they call it "leatherette" but it's bleeding plastic).

What the hell is wrong with cloth !? It's light, it's cool and breathable, it's cheap, it's durable, it's the perfect car seat material. I've read that BMW puts cloth seats in its cars in Europe, it's only America where they think they have to play into this "leather is luxurious" retardation.

Bleck. I've also decided I don't really love the fat spongey M sports steering wheel. I'd rather have a thin hard one with molded grip. Like, say, the one in my beloved Prelude. It makes you feel more intimately connected with the road, and I like being able to get my whole hand around it.

Anyhoo, the only big decision left for me is manual or automatic (steptronic). Normally I would definitely go manual, but the steptronic is pretty damn good and the manual-mode control on it still makes you feel somewhat connected

04-02-09 | Too much

ZOMG there are way too many papers on image and video these days. I just found this huge dump of ICIP papers. I just downloaded them all. There are 5000 papers taking 2 GB. That's just ICIP 2002-2008.

See for example : ICIP 2008

Oh, and of course in typical fuckup style, all the file names are like "001037.pdf" ; Awesome. I may have to write a problem to parse PDF and try to find the title and stick it in the file name.

The bad thing about Wikipedia is not that it's inaccurate (as retards often joke). It's that democracy and the rule of facts makes it very boring. Any little bits of humor or commentary in the articles are stripped out because they aren't directly factually backed. I'd far rather read something that's full of exaggeration and distortion written by someone amusing who's trying to make a point.

And really, presenting just the facts is its own kind of distortion. So much of history is subtlety and innuendo that some winking allusions can give you a better picture.

Wouter mentioned a while ago that used Porsches were getting really cheap with this economy. I glanced at Cayman S listings and thought "meh yeah he's right but not hugely". Then I looked at 911 listings. ZOMG. You can get a 2006 911 S for under $40k. That's 50% depreciation in two years. High end Mercs have fallen similarly. The more exclusive high end cars like Ferraris and such haven't fallen so much, but they are way more available than they would be in a tougher market (I mentioned before that the Nissan GT-R is available for below MSRP, which I'm sure is pissing off dealers that hoped to sell it for way over MSRP).

So anyway, if you've always wanted a 911, it's a great time to get one. I've always heard they're really unpleasant as a rush hour daily driver, so I have to wait until I have a house and a driveway so it can be my second car.

I'm done with old apartments. As much as I think our place is lovely and charming, I'm sick of the quirks, the floors that creak and are thing as cardboard, the dust everywhere, vents that don't work, windows that don't open or close or keep out the cold or wind, shitty kitchens and baths, etc. I hate mid-century American shit too, so the only option is to go brand new. I'm sick of getting ill from mold and dust and lead and whatnot in these dirty old places.

The rent for homes vs. apartments in Seattle is totally out of whack. You can rent a whole house on Cap Hill for $2000/mo, one of the big beautiful homes on a really nice street. Decent apartments are like $1500/mo. WTF, that's completely out of whack. I was thinking I'd like to get a top floor apartment (or rent a condo) in a new building. That would give me views and a deck and high levels of swank, but they're like $2500+ which is pretty nuts given our economy and the low price of whole fucking houses.

I've bought a bunch of things recently and I'm super unhappy with all of them.

Levi's 501's : the fuckers have completely redesigned the 501. Of course they still call it the "501" because 501 is a *brand* not a *model*. In fact ironically they now label them "The Original 501" when they are far from that. The actual original 501 (well, not the original 501, but the one that was pretty constant from the early 80's to late 90's) was marvelous and fit me perfectly. The only problem I have with the older 501's is that they take *forever* to break in. I've got a pair of the real shrink-to-fit 501's that I've had for over a year and they still haven't broken in. The new "original" 501 has the more modern thin denim that you don't have to break in, but they also totally changed the cut and it fits weird on me. Among other things they now have false waist sizing to mollify people who claim to be thinner than they really are (this is standard practice amongst most fuck-tard fashion brands, they label the waist a 34 but it's really more like a 36).

I bought a down comforter from Pacific Coast Feather Co . They are offering 20% off the allergen-free stuff right now so I thought WTF, this company is supposed to be the best. I have visions of a white bedroom, with morning light pouring through the windows, waking up all nestled in the fluffy down comforter. The comforter is shit. It's made of that fucking awful modern synthetic material that's all shiny and crinkly, it feels gross and makes weird noises when you rub on it. God I despise modern synthetic materials with a passion. I want cotton and tweed and silk and leather and wood and dirt and stone and flesh.

Which reminds me of my many recent failures trying to buy new jackets. In SF I went shopping one day because I need a decent jacket and don't have a single one; I went to a bunch of fancy stores, Nordstrom, Macy's, Kenneth Cole, some other little ones. Every single fucking jacket is synethic crinkly shiny awful shit. I don't mean blazers, I mean jackets that zip up. Everything from fucking Kenneth Cole, Banana Republic, J Crew etc etc it's all fucking polyster nylon shiny crinkly shit.

I long ago made a rule for myself to never buy clothes online, because they always are horrible in some way you can't anticipate, either the material is shit or they don't fit or they have some weird logo you couldn't see in the previews. But I do sometimes try to buy something that's just another one of something I already have. Oh no! That doesn't work either because the fuckers keep changing things without changing the name.

04-02-09 | Finance

I'm thinking about getting my cash back into stocks (long). I'll probably dollar-cost in slowly over the next 12 months or so. I'm kind of tempted to go big on Citi and GE. They're trading so low that there's a high probability of huge returns (100%+ easily). I really don't believe there's a huge risk either, since the government has shown no signs of bailout restraint yet, and they would get further prop-ups and tax favors and everything else the government can do to keep them from failing. It really seems like a government-backed gamble which is a wonderful place to be (and the same way that the mortgage industry made so much money all those years).

In contrast, something like LVS (Sands) is not such a desirable gamble IMO. It does have a nice huge possible upside (casinos + crazy chinese gamblers = megabucks), but it's not going to be bailed out by the government the same way something like Citi would, so it's not a free ride gamble - it really might go to zero and go bankrupt.

I have some slightly different views than many pundits on the economy. For one thing I believe the sickness extends far beyond just the finance and real estate sectors. I believe 99% of our entire country is broken. Our health care costs are too high, our workforce is too unskilled, we haven't been spending on R&D or infrastructure or education, our immigration policy is too draconian, too much of our economy is in services and consumer spending.

On the other hand I disagree completely with the people who claim that the US Financial Behemoth will never be the same, or that it's gone for good or whatever all the silly pundits are saying. In fact I would predict the exact opposite - the US Financial sector will be back to making huge risky gambles just as fast as they possibly can, and eventually our government will help them do so again. It might take a little while for people to forget, but there's one thing you can be sure of : once a problem is fixed, everyone forgets about it and starts removing the fixes and things that would prevent it from happening again. So the reckless financiers before the Great Depression led to lots of laws and regulations preventing it from happening again, with limits of bank sizes, minimums of liquid capital, regulators placed in the banks to watch their actions, limits of what different types of financial services could be mixed. But in the good times those rules were eroded little by little, until Clinton and Bush nearly completely eliminated them. And the exact same thing will happen again. Granted, it took almost 50 years for it to happen the first time, but we are far more flighty and apathetic now than people were in the last century. All we need is a new crisis to forget about this one.

American crisis response always progresses something like this :

Severe harmful overreaction that doesn't really fix much

Scapegoating of some minor players

Forming of committees to investigate and lots of announcements that lead to nothing

Things mostly get better and a new crisis comes along

Giving in to lobbyists that want rules relaxed so they can make more money

Letting it happen again

So shall it be with banking, terrorism, etc.

04-01-09 | Brain Vomit

Still sick but I think it's almost over. You can tell I'm home sick and bored out of my mind because I'm writing a ton of nonsense.

I want to go shopping here, but you have to fucking pay to park in the downtown shopping area. I'm not fucking paying to park just so that I can have the privilege of giving people my money. It's like if you had to pay $5 just to browse Amazon. (privilege and cartilage really mess me up). I don't understand how people think they can put up so many barriers to me spending money and still get it.

It's not surprising that Freeman Dyson has gone mad. Almost every great scientist does go mad in their old age (in fact the big surprise is the very rare case that they *don't*). Obviously all old people lose their grip on reality a bit, but geniuses and scientists seem to lose it more spectacularly. (a few just off the top of my head : Prigogine, Penrose, Pauling, I'm sure there are better examples).

Also Dyson's argument about global warming is just foolish. It's a sort of common left-brain physicist reaction to want to have all the information and completely understand all the causes & effects before you take action, but that's not always possible. With something complex like climate or politics, you may *never* get all the information, because the system keeps changing faster than you can measure it. The correct thing to do is consider the EV of the various possibilities :

P = probability that CO2 is causing warming and we need to do something about it

(when I say "warming is not happening" I mean it's not significantly caused by human actions or is not a problem,
obviously it is happening the only debate can be what's causing it and is it bad)

If we act :

EV(act) = P * EV( warming is happening and we act ) + (1-P) * EV( warming is not happening and we act )

If we don't act :

EV(don't act) = P * EV( warming is happening and we don't act ) + (1-P) * EV( warming is not happening and we don't act )

Let's put some sample numbers in to make this clearer :

EV( warming is not happening and we don't act ) is a no-op, so it's 0

EV( warming is not happening and we act ) is just the cost of acting for no reason, let's say it's -1.0

EV( warming is happening and we act ) is we do our best to stop warming, let's say it's -2.0

EV( warming is happening and we don't act ) is the disaster scenario, let's say it's -100.0

If we act :

EV(act) = P * ( -2 ) + (1-P) * ( -1 )

If we don't act :

EV(don't act) = P * ( -100 )

Equality at :

2P + (1-P) = 100*P

P = 1/99

So even if there's a 1% chance that warming is real, we're better off acting. The fact that we aren't sure of the cause & effects is somewhat irrelevant.

Surly Gourmand is a totally fucking awesome Seattle food blog. Holy crap I love this guy. I found it when searching for info about "Burning Beast" which I guess I missed but seems moderately rad. There's lots of funny little tidbits on Surly Gourmand, for example don't miss the comment at the end of this page . Ha! I like this this post too; hell it's all good, did I mention I love this guy? Here's a small taste of his well-salted genius :

Writing restaurant reviews is hard fucking work. “Cry me a river, asshole,” you might be thinking, and in a way you might be right, but in another, more accurate, way you'd be a dumbass.

03-31-09 | Taxes

I did my taxes this year with TaxACT . Mainly because I put it off too long to get a decent accountant, but also because they were so simple this year it seemed worth it to just do myself.

I liked TaxACT, I recommend it. I used TurboTax years ago and it was a bloody nightmare. By the time I was half way through I wished I had just used pen and paper. TurboTax was a weird annoying interface that didn't match the real form exactly and I couldn't figure out how its questions matched up to the data I had to enter.

TaxACT starts out with the same kind of annoying retarded interface where it just asks you questions; it tries to be like a "Wizard" for tax info, like "what kind of taxes do you want to do today?". Fortunately you can just close that down and go directly to the forms. It has all the forms with number entry boxes, and it does all the bits of copying numbers from one box to another and adding and subtracting and whatnot.

Sometimes when I do taxes by hand I feel like the IRS has created one of those math puzzles where you do a bunch of steps and always come up with the same answer, like :

enter your pre-tax W2 income in box 41
take the last digit and subtract it from nine and enter that in box 42
divide box 32 by box 33 and put the remainder in box 43
evaluate the gamma function of box 44
take the last three digits of box 44 and add them together, enter in box 45
copy box 41 to box 46
that's your income

anyway, TaxACT does all that bit for you.

Some caveats about it :

1. It claims to be free but it's not. I didn't need the premium version ($10) but if your taxes are complex at all (eg. itemized deductions) then you do. I did have to pay for direct deposit ($8) (it says free efile but doesn't mention you have to pay if you want the DD) and the California add-on ($14). Still cheaper than TurboTax.

2. Every time you try to enter a derived value it says "this is a computed value, do you wish to do the worksheet" , say yes and go do the worksheet. If you don't, it messes up its spreadsheet computation tables.

3. It can be a little hard to go "back" to forms you've previously done and want to edit. The best way I found is just to browse the index of all forms, and select them manually. Another trick is if you want to go to a worksheet that filled in a certain derived quantity, you can just try to edit the result and it will pop up the dialog mentioned in #2 and then you can say "take me to the worksheet".

Anyway, still not a huge advantage over just using pen and paper, and it does have the disadvantage that you have to copy in all the data off your 1099's and W2's and whatnot manually if you want to efile.

03-31-09 | Recent TV Reviews

I just started watching "Life on Mars" (BBC) with Alissa. It's decent so far. It is rather ham-handed with the whole is-it-real existential crisis. It also beats you over the head with the setting over and over just like the horrible "Mad Men" ; like oh look they're smoking how quaint, oh look there's an 8-track in the car, how cute. It would be so much better if they just told the story straightforwardly and the setting was just *there* without all the over-emphasis. Life on Mars wins over Mad Men just because it actually has stories - the Not sure if recommend yet. I think I would rather just see a good cop show rather than this self-conscious highly affected play on a cop show.

(BTW of course if you have to option between an American remake and the BBC original - always watch the original. The American formula is to basically steal BBC shows, put beautiful retarded actors in them that don't match the character, rip out everything that's real or honest or edgey or controversial at all, make them just like every other bland TV show, use tacky overly slick fancy production and lighting and makeup and everything).

I've been watching a bit of "Shameless". It's basically a Northern England soap opera, and I feel a bit dirty watching it, but it's very well made. It's got a great pace and rhythm to it and the characters are very lively. Recommended for losers like me who wish they were watching soaps but would never admit it.

Watched all of the first season of "That Mitchell and Webb Look". I must say watching it all in one go was not friendly to it. The running gags are very repetetive. The first episode I really liked and it just went downhill. Still a few bits really made me lose it. I love the realtor bit, and the natives in the gardening supply was laugh-out-loud gold.

I watched an episode and a half of "15 Storeys High" then gave up on it. I just don't get it. Is it supposed to be a comedy? I don't think I've seen a funny bit yet. It's just sort of weird and grim. I guess there are some bits that are obviously supposed to be funny, like when the guy doing the relaxation tapes gets mad and yells, but that was just so obvious and set up and not at all in context that it completely misses. Do not recommend.

Recent Movies (just the good ones) :

"Man on Wire" was beautiful, inspiring. I've always wanted to live more like that - living every day artistically, beautifully, doing something crazy and pointless and dramatic with a band of friends that are united by the sheer drive to be doing *something* this these sad boring lives of ours.

"Milk" was quite good. A bit heavy handed & cheezy at times (the flash back at the very end to the foreshadowing at the beginning was just inexcusable), but overall very moving, well acted, fun. It made me really miss San Francisco (we saw it before GDC). All the crazy fun people, the liberal politics, all the protests and parties in the streets.

03-31-09 | Recent Food Reviews

I've become a big fan of braising greens as I get older. The Trader Joe's braising green mix is reasonably priced and convenient, and unlike most bag salad products, it's not all rotten because braising greens hold up well. At Restaurant Zoe we had some collard greens that were almost delicious but in fact disgusting. They were smothered in so much butter, and it they were near room temperature so the butter was semi-solidified, so that it was almost like a quiche with just butter instead of milk and egg. If you're going to add bacon to your braising greens, don't be tempted to do it first. Yes, cooking things together is nice, but it's better to cook the bacon last and just sprinkle it on at plating so it stays crispy.

The best thing I ate in San Francisco during GDC was the bone marrow at Two. Bone marrow is a lot like butter-collards, it can be disgusting if it's too plain, because it is basically pure fat, and it does have a slightly weird vitamin-like bone taste. It's been trendy to cut it with a bit of persillade or gremolata or something like that, but at Two they serve it in a kind of reduced-broth onion gravy, which was perfect. It enhances the meatiness and tames the vomit factor, and provides some extra nice tasty bread dip.

(the Raviolo at Two was also excellent - a soft cooked egg yolk in a pasta wrapper with brown butter sauce, very simple but delicious. It would've been better with some crunchy pancetta bits sprinkled on top. That also continues a trend I've seen a lot recently of really interesting and well made delicious appetizers, leading up to rather boring and lazily made main courses.)

03-31-09 | GDC Middleware

There's a game engine for every day of the week now. Personally I don't think big game engines is the way to go. It just puts you as a developer too much at the mercy of the engine developer, and if it has major flaws that you don't like you're screwed. I'd much rather license lots of little pieces I can tack together as I see fit.

SpeedTree's tools get better and better but their runtime is still not awesome. That seems to be a pretty common thread. It's kind of weird because for me the tool is the hard part that I never want to do. Hell I could write the runtime for *all* these middlewares if they would write the tools and the do the sales and support. Some other kind of similar ones :

Fork Particle Middleware has a pretty nice tool. It's funny I was talking to Sean at the show about what other middlewares RAD might do someday and I mentioned particles. We figured you'd want to have a nice tool for artists where they could compose arbitrary particle systems, then the ideal thing would be to output a sort of "particle HLSL" for each system, then you could compile that to CPU code or GPU code or SPU code. The idea there is that all the toggles hierarchy and such that the artists set up, you don't have in the code. That is, you want to avoid winding up with code that's like :

if ( m_particleSystemDef->m_doSpin )
... more ifs for each possible effect ...

instead you just compile down optimized code for each system type. You also would want to automatically provide particle LOD and system scalability so it can target different machine capabilities.

Anyway, I doubt it's a good RAD product because it's too tool and artist heavy, it's not a big enough piece of pure technology. The Fork demos look pretty super-duper asstastic, like shockingly bad, but I don't know if that's just because they have bad artists setting it up or because their tech actually sucks. I do think there's a lot of value in actually making an impressive good looking demo - it proves that it's at least possible to do so with your system.

Allegorithmic Substance was at the Intel booth showing their procedural texture middleware. This is another one that I had the idea of doing some day maybe, so I wanted to see their action. Procedural texturing is obviously compelling in the future era of 8+ CPU cores and SVT infinite textures and so on. And you clearly need a nice artist-driven tool to define them.

Allegorithmic has a really super nice polished tool for writing shaders. I have no idea how fast their runtime is; I tried to ask some questions about how they run the shaders and the guy demoing didn't seem to be an actual programmer who knew WTF was going on. Again just like particles you would want to be running code generation so that you don't have like 100 per-texel branches for all the options. In fact, this is really just like rasterization in a software rasterizer like Pixo. The procedural texture code is just like a pixel shader and you want to output some kind of HLSL and have it compiled to a CPU/GPU/SPU shader.

I also don't think their way of defining shaders is right. You want something that is intuitive to artists and easy for them to tweak visually. You don't want to have to hire someone who is a procedural texture specialist. The Allegorithm shaders are just like Renderman shaders (or as Sean pointed out - like the stuff that demo coders do for 64k demos). You get a bunch of functions that you can compose and chain together and you tweak them to make it look like something. For example you can take Perlin Noise and apply curves and powers to it and threshold it and all that kind of stuff. Or you take a "mother wavelet" shape kernel and tile it, randomly rotate and offset it, etc.

That stuff is powerful and all, but it's just not intuitive and hard to tweak and not good for artists. If I was doing procedural texturing it would be example-driven with little bits of functional stuff. Some stuff you could do is all the shape-by-example synthesis stuff. You can do "detail textures" the correct way by using different tiling textures at different frequencies for the different wavelet levels of the output. You can use multiple tiling source textures and randomly compose them using Perlin Noise or an artist supplied blend texture. You can do things like the old "splatting" technique from Surreal with blend-alpha channels in tiling textures. You could even do things like Penrose Tiles where you have the artists paint the tiles and then you can create infinite non-repeating tilings. Or you can use sample textures to create tiles automatically like the Hoppe lapped texturing stuff. etc.

It just seems like the Allegorithmic tool is aimed at an audience that doesn't exist. It's too technical for real game artists to use. But if you're going to have a technical artist / programmer write the procedural textures, they would rather just have a text HLSL type language like Renderman. And having text shaders like that is much better because it's easier to share them on the web. In fact, the only way a tool like this could ever really take off is if a community develops and they post shaders on the web and share them, because writing all the shaders yourself is too much work. (in raytracing there are tons of prewritten shaders that are free or sold in commercial procedural texture kits).

Actually what you want from the tool is just a general attribute plug editor. You want to write text shaders like Renderman but have them take various parameters as scalars, colors, or images. Then you want to expose those to the artists with nice GUI tools and show them how they affect the shader with realtime preview. Something like :

input parameter brick_color : type color;
input parameter mortar_color : type color;

color diffuse_shader(vec2 uv, vec3 worldpos)
    ... do shader maths using uv, brick_color, mortar_color ...

In fact having a generic parameter editor set is a handy thing that all studios should have and everyone reinvents, but again it's too small of a piece to sell.

03-31-09 | Spring Cleaning and Mucus

Living up here reconnects me with the idea of "Spring Cleaning". Living in the warm south it never made any sense - why pick Spring to do a big cleaning once a year? Just do little cleaning all the time. (actually I'd like to have a schedule of one major cleaning item to do each month, like Jan - launder drapes, Feb - remove grates and dust inside air vents, Mar - move appliances and clean behind them, etc).

Anyway, up here it's cold enough that you close everything up for the winter, and I guess in the really cold places that's even more true. Heck in the old days when Spring Cleaning became a custom, they were still putting boards over windows and bringing livestock inside for the winter to stay warm. You would close up for the winter and swaddle in blankets and live in the stale musty air and dead skin and dust.

Here it's getting stale and dusty inside. I prefer to live with my windows open all the time. Sadly I haven't been able to do that for quite a while, since here it's cold and wet, and even in San Francisco if I opened my windows a pound of filth would come in the window every day. Anyway, I'm looking forward to when this cold and gray and wet finally ends so I can open up all the windows and shake everything out.

As I've gotten older something has changed in my nose geometry. As a child I always had a leaky nose, causing me to frequently wipe on my sleeves and t-shirt when I couldn't find a kleenex fast enough. Finally I gave in and started carrying a handkercheif like my mom (which I always found disgusting). Gradually something in the cartilage has changed, and now the snot drips down my throat instead of out the front (maybe this is just because it's constantly plugged with boogers). Anyway, it now makes me spit up loogies instead of dripping snot, which is also gross, but is better I guess because I can at least control it a bit and decide when to let it out.

Anyway, while my constant spitting of loogies is gross, and I sympathize with spitters, there are some spitting habits that people have which I find completely repellant and mind boggling :

1. The constant hocking / sniffling. This guy has got a mucus problem but for some reason is not going ahead and expelling the offending matter. He just keeps hocking or snorting over and over without ever spitting. This is sometimes done by people who were told by their mother not to spit, so they're trying to have good manners - well bud, hocking over and over is much worse than just doing it once and getting the spit out.

2. The gratuitously thorough hock. This guy reaches deep and grunts and rumbles to stir up mucus from deep inside then does a massive hock and spits out a liter of saliva. If you really needed to hock that hard to get the stuff up then you really didn't need to spit that bad in the first place. It's like they do it as a show intentionally to hock as loud as possible, as if it's manly or tough or something. This is usually done by working class males; it's sometimes used as punctuation in conversation.

3. The dripper. This guy hocks and then doesn't spit a tight wad with force the way you're supposed to, he just sort of leans over and opens his mouth and lets it drip out. Most of it comes out quickly, but there's usually a long trail of sticky slime that hangs off his mouth for many seconds, and he just sits there and lets it slow stretch and finally detach. Come on man, first of all spit it harder so that doesn't happen, but if you get a hanging trail, use your hand to detach it or something. This is often done by smokers or tobacco chewers.

03-30-09 | Price Comparison Ruins Quality

Online price comparison is very harmful.

Meh I'm bored of this rant but you can fill in the blanks cuz it's obvious. Here are some key words for you :

Expedia - airlines competing only on price, Nextag, eHealthInsurance / Progressive / Geico - insurance companies competing only on price.

Obviously quality goes to shit.

03-28-09 | Old Man Comfort

When I was young I was really nervous around people or in new situations; I wanted to show that I knew how to handle myself, I never wanted to make a mistake, I wanted to prove that I deserved to be with the A-list. It made me a total loser, really self conscious, condescending, stiff, and just awkward and antsy. I was also a huge a flaker and would frequently bail on social situations or just go very briefly.

I was always jealous of the older men who just seemed comfortable in any situation. They were usually somewhat wealthy, but not rich, relaxed, maybe a bit fat, graying, balding, but just confident and at ease. While I'm still certainly more like the youthful me, I'm starting to get a bit of the Old Man Comfort.

When I was a kid I thought that I would become comfortable over time once I learned how to act in all those situations, how to interact with the bell hops, how to act when you order wine, etc. etc. Now I know that actually learning how to act has nothing to do with it. It's more just about not caring. As you get older you care less and less what people think and you know it's just not a big deal if you fuck up. For me there have been two big factors :

1. I've given up on the hope of the Ivory Tower. I used to imagine that there were amazing people somewhere, smart, beautiful, fun, having great conversation, great parties, starting businesses together, making art, cooking ,traveling. I wanted to be one of them, and I wanted to prove that I deserved it. I always thought most people were shit, but I idolized certain heroes in my fields that I thought were just total rock stars and I wanted to act cool around them and get their approval.

Now I no longer believe in that. Yes, there absolutely are different classes of people, and yes there is a AAA group that is way better than the rest in most ways. But they're shit too in their own way and not worth impressing. Basically everyone is shit and there's no magic circle you can get admittance to so you don't need to try.

2. I'm no longer as interesting to myself alone as I used to be. When I was young, hanging out with other people was borderline excruciating because you'd just be having some horrible boring conversation and never getting into the real issues; even if you did find a smart person to talk to, you'd just spend the whole time in misunderstandings and talking around the same point. On the other hand, if I bailed out on that social situation, I could wander the streets and look at the beautiful plants and compose poems in my head while isolating various of my senses one by one and focusing on them to turn up their intensity. I might go home and work on a programming project for a while, or crack a notebook and work on some physics theorems, or read some textbooks by brilliant people. Every day I spent alone was full of activity and thought and projects and learning and excitement. Pretty much any minute of that spent with someone else was a loss, and it made me feel really antsy and impatient with people and want to rush through conversations and cut them off as quickly as possible.

While that's all still true to some extent, the big thing that's changed is not that I've discovered the value of other human's conversations (no, they're still as useless as ever). The difference is the value of my own alone time has plummetted, bringing them closer to parity. I'm now lazy and tired and bored of myself, I have lots of project ideas but I no longer have the enthusiasm to do them justice.

03-28-09 | GDC

... was really rough. I didn't get to any talks so I don't have anything good to tell you. I got to walk around the floor a bit, but didn't see anything interesting really. There is a ton of middleware out there - a procedural texture product (Allegorithm or something), FMOD, SpeedTree, a particle system middleware - and all of them have really really polished tools, super nice UIs. There are also just a ton of game engines now, some of them have very flashy demos.

I miss San Francisco. Even just flying in, I looked out the window and could make out the weird shape of Point Reyes and thought of all the great biking out there, the fun people who go swim Bass Lake, the sun shine, the aromatic smell of all the sage and bay laurels. The city has such a happy vibe to me, so many different types of people all jammed together and getting along, hobbos, hippies and hipsters. People biking around, sitting in the park, everyone seems to be energetic and full of life.

It was interesting seeing the break down and set up. It's amazing how fast they do it. Literally the second the 3:00 closing bell is chimed on the last day, the union labor guys start pouring in and picking up carpet. The RAD Veterans immediately started tearing apart the booth and boxing it up so it would be ready and they could get out of there. In a few hours the whole place was torn down.

Also, a lot of people have mentioned "OnLive" in their GDC blogs. It's 100% retarded and I don't want to hear about it any more. I would bet any amount of money against them ever making any money.

... and I got the GDC Disease from shaking hands with weirdos from around the World. As soon as I got home I was hit with a brutal cold, headaches and mucus.

03-24-09 | Cute but retarded

There's an article in last Sunday's New York Times called "Try, Try Again, Or Maybe Not" that I think is a great example. It's sort of cute, it comes across as smug, it interviews experts. And it's just completely wrong.

The core point goes like this : 22% of venture capitalists succeed. Those who have previously succeeded have a 34% success rate. Those who have previous failed have a 23% success rate. The conclusion is "you don't learn anything from failure, the success rate is basically the same".

Totally retarded. They completely miss the basic point that there are varying populations and the previous outcomes weight for the different populations. Furthermore the success rate of the group with experience is (0.22*34 + 0.78*23) which is clearly better.

Let's run some example numbers. Let's say the population is made up of good entrepreneurs and bad ones.

Initial population :

X good
(1-X) bad

success if good = P(G)
success if bad = P(B)

Initial success = X * P(G) + (1-X) * P(B) = 0.22

X * G + (1-X) * B = 0.22

In the successful group,

fraction successful Y = ( X * P(G) ) / 0.22

success rate = E + Y * P(G) + (1-Y) * P(B) = 0.34

E = additive benefit from experience

0.22 * E + ( X * P(G) ) * P(G) + (0.22 - ( X * P(G) )) * P(B) = 0.34 * 0.22

In the unsuccessful group :

fraction successful Z = ( X * (1 - P(G)) ) / 0.78

success rate = E + Z * P(G) + (1-Z) * P(B) = 0.23

0.78 * E + ( X * (1 - (G)) ) * (G) + (0.78 - ( X * (1 - (G)) )) * (B) = 0.23 * 0.78

This is three equations in 4 unknowns, so there is a variety of solutions.
X can be in [0,1]
For each X there is only one valid {G, B, E}

... meh I started to solve this but I got bored. The solution is something like G = 0.4, B = 0.1, E = 0.05.

Anyway it's absolutely clear that E is significantly greater than zero, and in fact the whole thesis of the article is wrong. Actually something the numbers are telling me that I think is perhaps more interesting is that the "Bad" population is *very* bad. The reason why the group that failed once is not too bad the second time is because there are still quite a few "Good" people in it that just got unlucky on their first roll, and everyone learned a bit from experience.

03-23-09 | A Better Log Window

I just put a LogWindow in cblib and I thought I'd make some notes about it. The reason it exists is that stdio on Windows is so ungodly slow. The console is actually a seperate process, and when you printf to stdout, it creates an interprocess communication packet and fires it over. I'm sure every programmer knows that doing lots of stdio printing can be the bottleneck in your app if you're not careful. We all take this for granted and take a lot of steps to avoid printing too often. For example my file copy percent display routines look like :

    int lastPercentShowed = -1;

    .. copy N more bytes ..
    int percentDone = (100 * bytesDone + (bytesTotal/2) ) / bytesTotal;
    if ( percentDone != lastPercentShowed )
        lastPercentShowed = percentDone;

where the whole purpose of lastPercentShowed is to avoid doing too many prints. (you could also use timers and check enough time has passed, etc).

But sometimes it's a pain to limit this kind of stuff at the high level. For example Oodle logs all the files it opens. Normally that's fine because you don't open files too often. But sometimes you do something that opens a ton of tiny files, and suddenly the printf to stdio becomes the bottleneck. Well you know what, that's retarded, it's just putting text on the screen, it shouldn't be such a huge perf hit.

So I just did my own Log Window.

It's just a simple Window on a thread. It runs its message pump for that Window on that thread (BTW to do this you must also Create the window on that thread - Windows automatically associate themselves with the thread they are made on, unless you do AttachThreadInput which you don't want to do).

The LogWindow::Puts from the main thread just uses my lock-free SPSC FIFO to send the string as a packet to the LogWindow thread. That makes it nice and nonblocking and fast for the main thread. (I just duplicate the string to make it safe to send to the thread).

So that's all nice and simple and fast and the main thread can just fire tons of logs and not worry about it. The other big speed improvement is in the LogWindow update.

The LogWindow thread sleeps when it has no windows messages. It needs to wake up occasionally if it got a mouse click or a paint that it needs to respond to. It also needs to pull new strings off the FIFO and add them to its list. When it does that, it can just keep pulling off the FIFO while it's not empty - it doesn't cause a redraw for every new string it gets, it only redraws once for a whole bunch of strings. This is much neater, if the main thread is firing tons of strings at it really quick, it will wind up just grabbing them all and doing one big step instead of tons of little ones.

The LogWindow thread also needs to wake up when there are new messages in the FIFO even if it hasn't gotten a paint message. I guess I should do that with an Event signal, but right now I'm just using a timer. The timer just wakes it up and makes it pull the fifo. Yeah in fact I'll switch that to an Event right now and use "MsgWaitForMultipleObjectsEx" that lets you wait on either an Event or a Windows HWND Message.

Though actually they optimize slightly different cases. The Event is better if you log very rarely or with big gaps, because it lets the LogWindow completely sleep when it's not being updated (with the Timer it still wakes up on the timer interval all the time and costs you some thread switch penalty even though it does nothing). The Timer is better if you are logging very heavily, because just setting the Event for each printf is not at all free. Maybe there's a way to get the best of both worlds, but really either one is miles ahead of using a standard console.

03-23-09 | Oodle GDI and Strings

Some random Oodle thoughts/questions :

My ThreadViewer is currently using GDI. It's pretty good, I like it. I did originally because I consider it a favor to the user. There are some nice things about using GDI instead of a 3D API - it plays nicer like a normal windows app, you know it gets Paint messages with rects and only repaints what it needs to, it can be dragged around and respects the user's paint while dragging or not request, and of course it doesn't gobble CPU when it's just sitting there. And of course it doesn't interfere with other 3D apps using the same device driver and memory and whatnot.

There are some disadvantages though. GDI is butt slow. If I draw much at all it really dogs. It's possible I could speed that up. For example I'm drawing individual lines and it might be faster to draw line-lists. I also wonder if it would be a win to batch up states. Right now I do SetPen/SetBrush and then draw, over and over, I could batch up draws with the same settings to avoid state changes. Another disadvantage is fancier blend modes and such are tricky or slow with GDI.

One option is to do my own 2d drawing by locking a DIB and using my own rasterizer. Probably crazy talk. The other option is just to be rude and use 3d.

The issue of strings in Oodle is quite interesting. The way you refer to resources is by name (string). Obviously the file names that I load have to be strings because that's what the OS uses. The main data structures that map resources to files are thus big string->string maps. (at some point I'm going to rewrite this to make it a bidirectional map, a database that can go resource->file or file->resource, which is just string<->string with hashes both ways).

Currently what I have is a ref-counted string that can optionally also just be a plain C-string. It's a 4-byte pointer and I stuff a flag in the bottom bit to indicate if it's got a ref-count or not. If not, it's assumed to point at const memory. If it does have a ref count, the count is right before the string data. Thus getting to the characters is always fast. I also almost always pass around a 4-byte hash with the string, which lets me avoid doing string compares in almost all cases. I'm usually passing around 8-byte (64 bit) "HashedString" objects. I only actually do a string compare if the hashes match but the pointers don't, which in my world is basically never. This is very good for the cache because it means you never actually follow the pointer to the string data.

One problem with this is that the strings can still wind up taking a lot of memory. The most obvious solution to that we've discussed before here. It's to explicitly use {Path/Name} instead of {Full Path}. That is, break off the directory from the file name and share the directory string for all files in the same dir. That would definitely be a big win and is one option.

Another option is to have a "Final mode" where the strings go away completely. The 4-byte hash would be kept, and instead of a 4-byte pointer to string data there would be a uniquifier to break hash ties. In order to make the right uniquifier you would have to have a full list of all the strings used in the game, which you of course do have with Oodle if your game is done. Then the uniquifier has to get baked into all the files, so there would have to be a "destring" pass over all the data that may or may not be reversible. This could be made relatively transparent to the client. One annoyance we had with this kind of thing at Oddworld is that once you bake out the strings, if you have any errors you get messages like "Resource 0x57EA10BF failed to load!". Still this would make the super-low-memory console wonks happy.

Another option is to combine the strings very aggressively. I already am running all the strings through a single global string table (the Game Dictionary that has the resource<->file map also makes all strings unique - this is a big win with the refcounted string - whenever you read a string from a file, rather than allocating a new buffer or that string you see if you have it already and just use the one you already have and bump the refcount). I'm not actually enforcing that to be required, but a lot of possibilities open up if you do require that.

For one thing, a "string" can just become an index or handle to the global string table. I almost never care about the contents of the string, I just use them as a key to find things, so if the char data is converted to a unique index, I can just compare index equality. Then, that index in the string table need not point at plain old char data. For example it could be an index to a leaf of a Trie. Each string gets added to the Trie and only unshared parts of the string make new chars. Each unique string is a leaf, those get labels and that's the handle that is given back to index the global string table.

Another option would be just some sort of LZ type compression. Some strings get full data, but other strings are just prefix matches, like "use the first N bytes from string M and then add these chars".

I think in the end I'll probably have to do the zero-strings option to please the masses. It also has the advantage of making your memory use more controlled and calculable. Like, you use 32 bytes per resource, not 32 bytes + enough to hold the strings whatever that is.

03-23-09 | Consumer Reports

I broke down and bought a Consumer Reports membership to check up on cars. It's reasonably cheap, you can get a $5.95 for one month, but of course you have to remember to cancel and there are many reports of that being a pain and then contuining to charge you.

More importantly it's just crap. It's worthless. There's not really much on there that you can't find for free around the web. The only thing of value at all is the reliability survey, the actual articles are pure piss. I was actually shocked at the lack of information, I kept clicking around the web site trying to find the information that I assumed I was somehow not seeing. And with the growth of free sites like TrueDelta and even just the forums of places like Edmunds, you can pretty much get the reliability information elsewhere.

I mentioned to Ryan this concept : I think you can trust car owners complaining about their cars. A lot of stuff you see on web forums is just whiners and cranks, but people are so biased to defend their purchase, after spending $20k or more on a car they really don't want to admit that they fucked up, so when you see someone say that they hate their car and regret buying it, there must be real problems.

03-23-09 | Tech Image Series 5

This time we see some error delta images. These are created by taking an image, compressing, decompressing, then subtracting from the original, and finally multiplying up by some constant so that you can actually see the errors (normally there are values like 0,1,2,-1,-2, so they would just appear black and not visible to the human eye).

Wavelet :

Lapped DCT :

03-23-09 | Neighborhood Spring

The lovely Big Picture series about spring reminded me that I've really enjoyed all the daffodils and crocuses that have been blooming here of late. It certainly doesn't feel like spring here - it's still cold and gray - but the plants seem to think it is. I guess daphodils and crocus are classic northern European flowers, and Seattle is a lot like northern Europe; it's different than the desert wild flowers of spring I'm used to in Texas and Southern California, like the poppies, bluebonnets, indian paint, all that kind of stuff.

Anyway, I took some pictures walking around the neighborhood the other day. I think there's a very modest beauty to the little patches of flowers.

Also, this is right near us and always makes me laugh :

03-22-09 | Intelligent Hedonism

Often in life you find that really stupid people actually basically do the right thing, then semi-intelligent people reason about things but get it all wrong and wind up doing totally the wrong thing, and then the truly intelligent do the same thing as the dummies. You see this in poker of course a ton, people who just "bet when they have a hand" are actually sort of doing the right thing even though they have no concept of what's going on. Then semi-intelligent people start thinking all these cock-eyed things like "wait that makes no sense, the goal is to maximize EV weighted by the probability of each outcome, so let me estimate and do some mental math..." and they totally cock it all up. Another example I believe is the pursuit of base physical pleasures.

The most animal of pleasures are the greatest. Animal physical and mental pleasures are our reward for getting through this life of ours. The physical pleasures are the taste of food, the glow of exercise, the tingle of drugs, the haze of booze. The base mental pleasures are things like ego boosts, the feeling of acceptance by friends or peers, the feeling of being loved or wanted.

Now, the dumb obviously have no problem accepting the base pleasures. But they get it all wrong. They do it vulgarly, without discretion or sense. For example the dumb boys who chase every bit of tail in the bars. Yes, I applaud them for seeking out the wonderful physical pleasure of sex, but they make it vile, they make it unpleasant for the good girls to participate, and they cheapen the whole experience. It should be subtle, full of anticipation and hints. Similarly the dumb fatties all over the mid west. Yes, I agree, food is delicious, but they do it all wrong, just stuffing their gobs full of cheap filth all the time. Better to eat less and seek out new exciting pleasures all the time instead of just trying to get more and more of the same thing.

Intelligent Hedonism is about pursuing the physical pleasures in a smart way. Not over-indulging, not over-using one particular pleasure so that you tire of it, not pursuing one when it's not wanted, and not just doing it for a fix. The Intelligent Hedonist is always striving to find new pleasures, and to enhance an experience. You might not get it very often, but when you do it's really good.

It can be simple things - like the smell of the air after a rain, or seeing a sunrise from the top of a hill - those are the moments you stop and savor, and you might spend days or weeks working to set them up.

The Intelligent Hedonist doesn't just throw back wine to get drunk. They enjoy every moment. First the joy of the hunt - reading, searching, shopping, learning. Then just touching the bottle and seeing the label - the moment of anticipation, wondering what it will be like. The ritual of the opening and pouring, the sound of the pop as the cork comes out and the glug glug of the first pour (at a restaurant the whole ritual of the taste test). Then the nose the taste the feeling. But not just wine - obviously a lot of people pretend to do this with wine now because TV and Movies have told them to. You do this with everything because you love it. With a chocolate bar, you admire the wrapper, the sound of it tearing open - hopefully it has a nice gold foil under the paper and you get to lovingly fold that back; then the color of it, the shape it's pressed, the feel of it just slightly melting in your fingers, the scent of the cocoa notes, then the feel in your mouth, on your tongue, the melting velvet, and then the taste.

The Intelligent Hedonist is not just a complainer or a crank. He/she doesn't just moan about the shitty restuarants or the bad movies. He seeks out the good stuff. His life is an endless quest for things that are made with care and skill.

A lot of semi-smart people see things like typical drug users and rightly think "that's gross, that's a cheap way to get a high, I don't need that, I'm above that". Well, sort of. You're right that they way the dumb hedonist is using drugs is wrong, but you're wrong in thinking that the highest goal is to avoid them. The same thing goes for casual physical sex, many semi-intelligent people see it and think of it as manipulative, degrading, cheap, unfullfilling, and thus swear off it completely and spurn all those who partake of it. Well, yes, again you are right that the way dumb people pursue that pleasure is in fact gross, but that's because they're doing it wrong.

03-22-09 | GDC 2009

... is gonna suck. Tons of people that I usually look forward to seeing aren't even going.

I'll be at the RAD booth much of the time, so stop by and say hi. I'm not really pushing my product (Oodle) yet, but there will be an early preview of it, and of course you can see all the other great RAD libraries (and a preview of Sean's Flash/GUI product too).

Obviously, the Larrabee talks are going to be awesome, but I haven't heard of anything else too exciting yet. I'll add to this post if I find anything worth going to.

03-21-09 | Actions for Outcomes


I wonder what the outcome would be if I yelled that out at the top of my lungs. Improvement or no?

03-21-09 | The Marvels of German Marketing

On all these car sites you constantly read about the "fine German engineering". What? I mean the stuff is really nice, I am very sympathetic to their aesthetic, and I love that they are driver-centric. But that's just the design choices. Their build quality is just awful. You get quotes like this :

"It is assembled with typical BMW care and craftsmanship with high quality materials used throughout"

the real marvel is the incredible German Marketing. Holy crap they have done a number on people.

You also get lots of auto writing that just doesn't make any sense at all :

"The acceleration, especially from a standing stop, is super responsive and yet restrained enough to delight."

What? In what world does restrained equal delight ? Oh yes, please restrain the acceleration more, oh how delightful. So a Hyundai Elantra makes you jizz in your pants?

03-21-09 | Search Awareness

It's very important to be aware of how your product name will work in search these days. If you name your band something like "Soap" you're never going to show up.

For example apparently there's a company called "Rad Technologies" in Hyderabad, and they posted a job listing on Oodle.com (a big classified site) , so if you search "Oodle Rad" you find them. There's also a Developer dot Ooodle dot com (for the Oodle classified site) which has an Oodle API and so on.

Anyway, the thing that's annoying me today is I'm trying to search for differences between Directx 8 and 9, and it's just impossible to search for anything related to DirectX any more.

What they should have done is made a different name for the runtime and the developer API. Call the runtime DirectX but call the developer interface apiX or whatever. That way when developers are talking about the internals, we can find each other.

03-20-09 | Car Buying Mentality

Car buying is one of those things where you can easily fall into the trap of worrying about what other people will think of it. I don't just mean buying a car to impress others, that's the obvious trivial way and not a big deal. The more subtle bad thought process is mentally justifying your purchase to others. People who buy a Honda Civic are preparing their rationale in their heads as they buy it, imagining conversations where they tell people "it's just so practical, it's very reliable, it's fast enough for me for now" or whatever.

I've always been biased against Porsches because they're underpowered for the money, and they are so often the car of middle aged mid-life crisis men. So what? Getting as much horsepower per dollar is not the goal. It's to get the car that you enjoy. In my head I can hear the taunts of the stupid public saying "oh but some dumb American car has way more power at half the price, how could you buy that?" and I prepare a response in my head. Fuck you, I don't even have to respond to you. Stop thinking of rationales.

In my head I keep hearing people say "why would you get a 1 series when you can get a 3 series for just a few k more?" Well, voices in my head, I'll tell you why - because I like it better. It's smaller, lighter, faster, stiffer, more nimble, more fun. But really the point is that I don't need to make reasons for you, voices in my head.

In the end there are just way too many conflicting factors to weight them sensibly against each other in any objective way. You just have to go with your instincts.

03-19-09 | How Not To Invest

You can learn a lot about how to invest by doing the opposite of me. Some of my guiding principles that have been hugely detrimental to me are :

"It's just a bubble - I'll stay out" . Sounds reasonable, but is quite silly. Bubbles are great opportunities to make a lot of money. Staying out because "it doesn't make any sense" or "it's rationally wrong" or "its fundamentally overvalued" is just shooting yourself in the foot. I've intentionally stayed out of tech stocks, real estate, gold, etc. because they are illogical bubbles.

"It's too late to play that now" . Over and over I've seen very obvious trends but figured I missed the good early chance to get in, so now I should pass it up. One example is bubbles, it's so obvious when they're happening, and I would think "oh, this is super obvious to everyone now, it's too late". No, it's not. These things take a long time and there's plenty of chance to get in late. There have been tons of obvious plays recently - like investing in China and India a few years ago - where I thought oh it's already obvious to everyone it's too late to get the good value.

"That's too obvious the market must have compensated for that already" . I always assume that the market is doing a good job of compensating for news and obvious trends. Like if some company announces an awesome new product and everyone loves it - the stock should shoot up based on that news and already have all the good news baked into the value, right? Well, no, not really. Things like "Halliburton will do well under Bush/Cheney" is just such an obvious play that I thought it would never work because the market had already compensated. Similarly stuff like shorting Home Depot in the recession.

"I should do what's been proven to work best over time" . In scientific studies of investment strategy, no portfolio strategy beats buy and hold in the long term (or if it does it's by a very small amount that's less than transaction cost). So you're "supposed" to just find broad market funds with minimum expensenses. Listening to that advice has been a huge mistake.

03-19-09 | Seriously?

My god I am so fucking sick of your retarded broken ass software. Fucking bill pay web site can't handle commas or dollar signs in the number entry boxes. WTF I just copy-pasted from the amount you are showing me is due into the payment box and you fucking freak out and can't handle it !? In fact, all web based UI can just bite me. Ten second stall outs to show me the next GUI box are not okay.

03-19-09 | Test Drives

So I went test driving with Ryan yesterday. Here's what we saw :

Nissan GT-R : it looks way bigger in person than it does in images, it's really a hulking beast. The cockpit is very tight, you really feel like you're wedged in, which I guess is good if you're gonna push it. Unfortunately this salesman was a complete dickwad retard and wouldn't let me drive it. WTF I'm not gonna buy it if I can't drive it. If I had felt how it moves and loved it I might have considered it, but it is rather impractical. The douche kept telling me about how rare it was and the premium on the price. Look dickwad, there are *tons* of GTR's for sale on Ebay right now, in fact it's easier to find than an Audi RS4 or a new M3 or a BMW 135.

Infiniti G37 : this is the "luxury" version of the Nissan 370, it's the same engine and chassis, just different body and nicer interior. I use quotes on luxury because there is not much luxury to be had here. Everything feels like a cheap Japanese car; actually it's worse than that, the interior is *way* worse than the interior of my Prelude, because the Infinity is all cheap plastic, but it tries to look fancy, so it's got this retarded plastic analog clock in the dash (LOL?) and these shiny fake-metal plastic pieces everywhere. The whole thing kind of reminds me of like a strip mall with roman columns in front of it. Anyway, the interior isn't that important to me, how does it drive? Meh. They only make the 4WD in auto so I tried the RWD manual. The shifter is okay (though we smoked the clutch up pretty good). It's definitely got speed, but you don't really feel it. The power delivery is weak at low revs, and the torque just isn't there when you want it. There is a huge excess of Infinitis right now so they can be had very cheap (perhaps below invoice).

Audi S5 : this thing really looks beautiful in person. The styling is sporty but subtle. It actually really reminds of the BMW styling, but just smoother and better. The interior is also (mostly) nicer than BMW, nicer feeling leather and better arm rests and just nicer trim all around. The only exception is the center console control stuff which is oddly bangled and blingy looking with all kinds of shiny metal and studded pieces that look like they belong on a rapper's necklace. Unfortunately the drive quality just sucks. It feels like a boat. It feels really huge and heavy, it's got a big powerful engine but it doesn't really feel fast. The steering was way too loose, by far the loosest of any car we drove that day. Throttle response was laggy. Visibility is very poor - the rear view mirror is tiny and the blind spots on the side are big. I really hate this trend in car styling where they make the body taper to the front so that the butt is really big, but they also make the roof taper down in the back, so that the rear window is just microscopic (the 370Z has the same problem, in fact a lot of these cars do). In the Audi you get a rear view camera to help you back up despite the bad window and the huge beastliness of this road-boat. Yuck. The sales guy told me there's a selective sport option that tightens up the feel but I don't believe him, and it would still be huge and heavy. It's a fucking V8 with 350 horsepower and it just felt slow. Plus it's a fucking Audi which is just one of the worst made cars around right now.

BMW : the dealer guy was nice and we drove a ton of cars and they were all quite pleasing. In all of them the interior is pretty nice and solid feeling. The steering controls are direct and responsive. Personally I'm not a big fan of the iDrive computer stuff, however it is better than the ghetto-iDrive-knockoffs that were in the Audi and Infiniti. It seems hard to find a car now without all kinds of unecessary gadgetry, and if you are going to have them the BMW gadgets seem to be the best. The non-iDrive interfaces to the stereo and such are awfully minimal, like it just looks like it's only partially finished, with buttons not even labelled and such.

335xi : this is the 4WD twin-turbo 3L V6 that doe 300 hp and 300 torque. This one was a little disappointing. Steering felt good, but throttle response was very laggy. I would jam on the gas and it would take a good "one missisipi" before the power kicked in. Not sure what the deal was, maybe the AWD was slowing it down? The RWD 135 we tried later had the same engine and didn't seem to suffer from the safe sluggishness. Also it was an automatic, even though I put it in sport mode it was hard to coax aggressive gearing out of it. It does feel a little on the big & heavy side to me, but suspension and steering were still very crisp. I'd like to try a manual but they didn't have any.

M3 : for the fuck of it I tried the new M3 with 4.0L V8 414 HP ; this one had the dual-clutch automatic / paddle shifter setup. The transmission was very quick and smooth. I'm sure I could drive much faster with this semi-automatic kind of setup than with a true manual, but I did miss the feeling of the manual a bit. I dunno if I'd get used to the semi-automatic over time. I think the steering wheel paddle shifts are a bit of a gimmick and not very practical. It's nearly impossible to keep your hands on them through a hard turn, which means you get stuck in a bad gear. Maybe on the track they would be useful. The engine sound in this thing is glorious, it's deep and rumbly and loud, it sounds like the beast it is, and it's got tons of power. It definitely does feel big and heavy, you have to muscle it around. The "M" button is indeed exciting. There's just something pleasing about flicking a switch to turn on the better mode (like the computer "turbo" buttons of old). It is kind of just a gimmick for me though since I think I would just drive it in sporty mode all the time. (with the "M" turned off it feels like a big heavy luxury sedan). While it was good, I didn't really love it. If you want the feel of an American muscle car but in a German luxury sedan, and some tight road feel - this is for you. But that's not really what I want. I want the feel of a go-kart, but with low end grunt, and some decent build quality and comfort. I do like the controls to turn traction control on and off and all that stuff.

135i : this is the same engine as the 335xi, but in a slightly smaller, lighter RWD package. Whoah, this was a revelation and probably our favorite car of the day. The rear seats are pretty tiny, there's no AWD. Again we were stuck with an automatic, but the semi-auto sport shift control thingy was pretty good. The front cabin is plenty spatious, you can sit up straight and spread out. It's got twin turbos like the 335, but you can't feel them kick in at all, there's no real power gap, it feels very smooth and always available.

While the 135 is a bit lighter, it's by no means light. It weighs 3,373 lbs , (vs 3,571 lbs for the 335i and 3,759 lbs for the 335xi). But it felt really nimble and stiff and peppy. It's got a slightly shorter wheelbase than the 3 series, which I think makes it feel tighter. On the down side, it is a new line which means it is somewhat rare and it doesn't look like prices are too great (I'm seeing it go for MSRP - close to $40k; I'd like to see it go for Invoice - close to $32k). Also because it's a new line there seem to be some 1 series reliability problems . Hrrumph.

The steering feel and response on all the BMWs was excellent. The double clutch automatic was superb, super quick and smooth, and the manual up-down control was okay.

Overall there wasn't anything I fell in love with. It was fun - my god, all these cars have tons of power, and many of them have great screaming engine sounds, but they just feel heavy and laggy. The 135 was the most pleasing and I think I could be happy buying that right now, but it's not totally ideal.

At the end of the day we got in Ryan's WRX and it reminded me what a nice car that is. Very light snappy feel, good control. If only it had a little more low end grunt and slightly more comfortable interior it would be pretty perfect. Eh, I also feel like you sit a bit too high and upright in the WRX, I prefer to be slung down in a cockpit a bit more. Fuck maybe I should just get one anyway.

As usual I was just shocked at how shitty the salesmen were. The BMW guy was nice, but only in a relative sense in that he didn't actively deter us from buying his cars. At every dealership we'd ask basic questions about the cars - what's the engine in this one, what transmission are available, what's the different between the sport package and the base, etc. and time after time we'd get "I don't know" or even worse - just completely wrong answers. We asked the Infiniti guy how much the G37 weighs. After a bit of hmm , err, he wanders around to read the labels on the car and announces 2400 pounds. I'm like, "umm , no, that's not right" , so he umms and errs around a while and announces "4600 pounds". Umm, no, I don't think so. (the real answer is 3,590 lb). I literally had to prompt the salesmen to give me brochures and business cards. You owe me 10% of your comission.

I also did not see any of the desperation that I hoped for. Despite constantly reading about how bad things are in the news, I have yet to see it. I found these nice graphs on the auto industry downturn that show just plummeting sales this year, and there are tons of photos around the net like this or this of huge lots of unsold cars. But I have yet to see big deals or much eagerness from dealers. Wouldn't you rather sell the cars for a tiny profit than just have them sit? It makes no sense to me. I have money and want to spend it, give me a bargain!

BTW fucking foot-pound vs. newton-meter for torque is a disaster. Part of the problem is the conversion is only a 1.35 multiplier so if someone says "300 torque" you can't just guess the units. That's probably 300 Nm = 222 Ft-lb. Of course there's also the horsepower fuckup; there's an "english" or "mechanical" horsepower and a metric horsepower which are off by a factor of 1.01, so at least that's not a huge difference but you have no idea what anyone is talking about. And then of course the standard horsepower that's quoted is "brake" horsepower (bhp) which is really sort of retarded, because it just measures the engine in isolation; what's really more useful is the effective or "wheel" horespower, but car makers never tell you that.

Some other things I might have a look at : previous gen (E46) M3's , they were a bit smaller and lighter, though I find the styling very generic and unexciting. Audi RS4 is very fast with good AWD, but it is an Audi, and very expensive, and awfully heavy and a huge gas guzzler. Lexus IS - the specs are good, but Lexus tends to make cars feel really mushy and boring, plus the manual is only in the 250. The advantage is that Lexus actually makes cars that don't fall apart.

03-17-09 | The Wrong Blame

As usual the retards that are finally up in arms about our financial disaster have started getting the blame all wrong.

The Daily Show segment about short sellers really pissed me off. (the fact that they mocked the random short seller guy and actually seriously listened to the crazy Overstock.com loony-toon guy is ridiculous). My god, short sellers don't bring down the companies. The companies are failing. Short sellers *might* occasionally help prick the bubble of inflated stocks that are going up purely based on momentum and collective belief in a fantasy.

There actually was a country that legislated that stocks could only go up. You were only allowed to sell stock for equal or greater value. (someone help me find a reference to that). Obviously that's retarded. Keeping bad companies afloat should not be our goal. Taking all the fluidity out of markets is not the goal either.

One bad aspect of the whole short sell / trading thing is how twitchy and over-reactive people have gotten. Not just day traders but professional brokers. Some good news comes in and stocks shoot up way past their real value. Bad news (like a bunch of shorts) comes in and stocks shoot way down. That's obviously bad.

If you're going to get mad about some practices in trading, there are plenty of things to be up in arms about. An obvious one is the intentional spreading of information (false or not) to affect stock prices, which has become quite standard and is borderline illegal and definitely unethical. The collusion at big banks between the investment bank part that offers a certain stock and the brokers and analysts that recommend that same stock is another.

Another very common practice to get mad about is the way the hedge funds stalk the major indexes. Quite a few of the large hedge funds now basically act as friction on the market. Any time the market wants to move anywhere, they take a piece. They act as an energy drag, sucking off a bit for themselves. This works basically because all the large mutual funds track certain indexes. A huge amount of the total stock ownership is through these large mutual funds. The indexes and the funds take some time to react to things, so when something happens, they have to announce it, and then it takes them a while to get things done. The mutual funds often have the problem that they have so much money to move that it takes them a long time to buy or sell the stock they're after.

The simplest and most obvious case occurs with the S&P 500 (or the DJIA). Whenever a stock is added or removed from the list of companies in the index, all the many huge funds that track that index must sell one stock and buy another. These huge quick computer-automated hedge funds jump in and buy up the stock that the funds need to get into, then sell it to them for a higher price. This is basically like if you walk into the grocery store and announce that you're buying apples, and some asshole runs over and buys all the apples and then offers to sell them to you for double the price.

These is just literally stealing money from long term investors. I'm sure there are lots of other practices like this that I don't understand, but plain old short selling is not one of them.

The other retarded fixation of the media was the whole corporate jet issue, and more generally just slight excesses of spending, like million dollar parties and whatnot. It's ridiculous on so many levels. First of all, it's a fucking drop in the bucket. It's not even a huge waste of money because it saves them time and whatnot. Second, the idea that the rich should stop spending lavishly or be embarassed about spending now is completely backward and will only make things worse if they tighten the purse strings. Third, the fuck-tard executives are getting paid way more than that in salary and bonuses and options; in fact there have been a few cases where some executive redecorated his office with a million dollar bidet and the media got all excited, so he just paid for it with his own salary. What? How is that better?

The recent AIG bonuses brings up a related point - yes, it's a bit ridiculous that they're paying out huge bonuses after receiving hundreds of billions of bailout money - but it's our own fucking fault. We just gave away a trillion dollars to all these fucking crooks with absolutely *ZERO* strings attached. No regulation, no requirement about how it be used. Of course they're just going to pocket it, why would they pump it into their failing business? That's what they're good at - giving themselves profits, that's their job to some extent. It's the fucking government's fault for giving out that money without regulation. Now congress and the media gets in a big tizzy and calls them in and says "hey, waaa you're not spending it how we wanted" and the executives are just like "meh, fuck you, you gave us the cash, I'm buying a new yacht".

I think the focus on the hedge funds like Bear Sterns or investment banks like Merrill is a bit out of place too. We should have just ignored them. It's perfectly fine for a hedge fund to take huge risks and sometimes lose on those bets - but we have to just let them fail! There is basically zero public interest in those companies. The invest the money of rich people and big institutions - they're not regulated and have got to just be allowed to fail. In the future we don't need tighter regulation of funds - we need to pledge to never bail them out. That might mean limitting how big they can be and also perhaps limiting how much regulated institutions can invest in them. Basically never want them to be "too big to fail".

The other myth that's being spread by the popular media is that all these poor homeowners are just victims in the crisis. Nonsense, in fact they're a large part of the problem. Now certainly a few people were victimized, given really bad mortgage deals that they didn't understand, and they just wanted a place to live and weren't speculating. But tons of people knew exactly what was going on. They were intentionally buying way out of their means because they wanted to get rich on real estate. They intentionally got neg-am loans because it allowed them to leverage up. They were speculating, they were watching TV shows about flipping properties. Most people knew we were in a bubble and kept buying anyway. In theory I don't blame them, they saw a profit opportunity and took it, but that doesn't mean we should now feel sorry for them or protect them or bail them out. However the reality is that US law has lots of favorable protections for mortgages, and speculating on investments that are protected by the government is definitely unethical and really should be illegal.

Basically my view is that there are two reasonable extremes : (1) completely free market with zero protections and little regulation, or (2) safe protected market with lots of regulation. Either one is okay (morally), but the thing that has gotten us in so much shit is our semi-protected semi-regulated market in which people can speculate and then have their risk covered by the government. That is fucked up. Also the illusion that things are regulated when they really aren't, and the ability of financial institutions to cross the dividing line between the safe market (in US law, a "bank" with FDIC insurace is supposed to be very safe and have plenty of capital and not take too much risk - similarly a mortgage is supposed to be a very safe type of loan that can be covered by the value of the property).

The big problem I see in general is that companies and individuals profits weren't tied to the deals they were making. Mortgage brokers for example were only motivated to do as many deals as possible, because they take a commision on each deal, and then just resell the actual mortgage, so they get none of the risk from the actual property. This gives them zero motivation to actual do a good deal or even turn down borrowers that are bad risks.

It seems like we need some kind of law to make profits more tied to actions. What we don't need is a bunch of micro-regulations about exactly what constitutes a safe mortgage or a good borrower or whatever. Unfortunately that's the way the US government tends to solve problems. It's much better to let the free market work and decide for itself who's a good borrower - the problem is just the mismatch of reward from actions.

Mortgages in particular seem pretty easy to fix. If mortgages could only be issued by institutions that backed the mortgage themself, it would all be fixed right there. Now they don't want to offer bad loans because they are backing the loan and they have to eat the default. Brokers no longer get paid just on volume, but rather from the actual return of the mortgages they offer. No more trading mortgages, no more government Mae subsidies backing the industry. Seems very simple and obvious and I don't really see a downside.

Another example is the credit rating companies. They have a complete conflict of interest and no motivation to rate things right. In fact their motivation is just to over-rate everything because that makes people happy and makes them issue more securities, so you get more business. The easy way to fix this is just to eliminate all of these made up abstract nominal credit ratings. Boom. Instead, the credit rating companies offer insurance on the securities. The value doesn't necessarily come in anyway *buying* that insurance, but simply seeing the price of it gives you the effective rating. That is, the price of the insurance on a given asset *is* the rating of how safe it is. If the rating is wrong, then the insurer can lose money because of the mistake. Thus they are motivated to give it the right rating. And it's in absolute directly measurable units - dollars.

Instead we'll probably have some fucking huge mess of laws about how the rating companies have to work, what exactly constitutes each of the rating scales, blah blah blah. That's terrible. Huge mess of regulation on top of a basically corrupt system is the wrong way to do things. Instead dig up the system and make it non-corrupt. Use the free market, but force the free market to work correctly - that is, make pricing honest and make the people taking the profit take the risk.

I like the idea of forcing people to price things as a way of enforcing honesty. Like if a trader recommends a certain stock - boom you have to buy it. If you recommend shit, you will lose money. Thus you are motivated to only recommend things that are actually good. You could make a law like anyone publicly recommending a stock has to buy it and hold it for at least a year.

03-16-09 | Windows NetWorking

Windows "File & Printer Sharing" Networking is really annoying me. It's one of the main culprits of long mystery stalls during boot. It also gives you the awful horrible stalls when you open Explorer for the first time.

What I would like is to be able to boot with Windows Networking completely disabled. Then when I choose at some later point I'd like to be able to turn it on. Can I do this?

The other thing that kills me is if I slip while typing in CMD and accidentally dir to one of my networked mapped drives, I just get a huge stall.

My Windows XP boot takes about 1 minute. I've been using "BootVis" and "BootLog XP" to check it out. BootVis is pretty cute, it makes nice graphs, but the actual important part of boot it just labels as "services" with no breakdown about *which* services. BootLog XP does a bit of a better job because it shows each DLL load and times each one, so you can make some educated guess about which services are taking all that time.

There's tons of little shit, okay, whatever. The AntiVir takes a lot of time, maybe 15 seconds total. Oh well, that's life.

csrss.exe is the next biggest time eater, it takes abour 10 seconds. Apparently most of that is because of loading the registry. I checked "pagedfrg" and my registry is not fragged at all.

The other big thing is svchost starting up the networking services which takes around 10 seconds.

03-16-09 | Temp Names

I made the classic error of actually trying to use the APIs you're supposed to use instead of just writing my own code from scratch.

First of all, the stdlib "tmpnam" is just horrifically broken in the MSVC CRT. I don't understand WTF they're thinking, but it's just awful in various ways :

1. It actually puts the files in the ROOT of your damn drive ! WTF.

2. It doesn't work in Vista at all (you can't fopen the name they give you back). Presumably because it's putting the file in the root and the app doesn't have write access to the root (?).

3. The names are awful, like "s38b" , usually with no extension at all. This makes it very hard to delete them safely when you get like 100,000 of them piled up in your fucking root.

Okay, okay, so we're on Windows we can use the Win32 functions. Well, on the plus side, GetTempPath seems to actually work, it gives you the dir specified by the "TMP" environment var (BTW if it can't find a temp setting, it falls back to the Windows directory which is kind of awful, but whatever, it at least finds the temp dir normally).

On the minus side, GetTempFileName is retardedly awfully bad.

If you just glance at the docs is seems like it's pretty reasonable. You stick 0 in for uUnique and rock on.

But then one day your app grinds to a complete halt because of GetTempFileName and you go "WTF?". Well guess what, GetTempFileName only actually makes up to 64k temp file names. And when it actually checks the name for existence and then tries again. This means that if you even get over 10k temp files in your temp dir, it can make names and check existence over and over before it finds an open slot. If you get more than 50k temp files, you app is basically dead.

Now obviously you don't want to have a ton of temp files hanging around, but Windows never cleans up your temp dirs so if you have some bugs at some point or even just other random broken apps, you can easily crud up your temp dir. And the prefix is only 3 characters so you can often run into other people's prefixes!

One improvement would be if they stuck the app's name onto the temp path and made a subdir, so that you at least got your own dir to play in so that one broken app couldn't mess up the whole system.

On the plus side they actually use the extension ".tmp" so you can go delete everything.

Anyhoo, this is all silly because we have this thing called "long file names" and we don't need to be using 4 characters of hex (!? WTF) which makes it so easy to get name collisions.

So I wrote my own; here's my current version of MakeTempFileName :

// stdc tmpnam seems to put files in the fucking root !?
void MakeTempFileName(char * into,int intoSize)
    // use better routines on windows

        char tempPath[MAX_PATH];

        static uint32 s_seqNum = 1;
        uint32 seqNum = s_seqNum;
        ++s_seqNum; // not thread safe, whatever
        // use tsc or something to get a real random int :
        Timer::tsc_type tsc = Timer::rdtsc();
        uint32 tscLow = (uint32)tsc;
        char tempFile[MAX_PATH];
        into[intoSize-1] = 0;
        // retry while this name exists :
        //  (would be very bad luck)
    } while( FileExists(into) );

yes, yes, this calls lots of cblib stuff.

03-16-09 | No place for money

My TIPS are down about 10% in the last year. All bonds fell a lot during the crash. I'm not sure exactly what the reason is, I guess corporate bonds lost value because there was a fear of defaults as people realized the ratings weren't what they were supposed to be. Also the yields have plummetted which has made them less desirable. I don't really get it though. I mean I guess even though the bonds are fixed income, the price for them is set by the market, so is prone to fluctuations of supply & demand.

Saving accounts are no good either. Here's the delightful ING reports :

Dec 30, 2008    Interest Rate Change to 2.472% (2.50% APY)
Jan 20, 2009    Interest Rate Change to 2.374% (2.40% APY)  
Feb  3, 2009    Interest Rate Change to 2.178% (2.20% APY)
Feb 18, 2009    Interest Rate Change to 1.835% (1.85% APY)  
Mar  3, 2009    Interest Rate Change to 1.638% (1.65% APY)  

Steadily plummeting to zero. Though really that's just an indicator of falling inflation. The real inflation rate is maybe something like the ING APY plus 2 or times 2 or something like that.

I think almost anything you do with your money is a loss. The reason the bank is giving you X% is because they think they can do better and make money off you. There's no free money in the world, the bank is basically investing for you and taking a big commision. Bank interest rates are almost always below inflation (the only exception is during brief periods when things haven't adjusted yet).

Anyway, since there's nowhere to save I figure I'll just invest in Hookers and Blow.

03-15-09 | Seattle Weather

The weather really hasn't been all that bad this winter. In fact since January 1st it's been sunny quite often, and it's absolutely gorgeous up here when it's clear, what with all the green and mountain views around.

Just recently another patch of the interminable drear has set in :


We've had a few bouts of hail and sleet and freezing rain and all that gunk. It's so much better if it just really snows, in fact I rather enjoy the snow if it would just dump and then get sunny again.

I like how the newspaper here does stuff like this :

03-15-09 | Hyundai Genesis

One thing I've been shocked by as I read about cars is just how fast they all are now. In the last 10 years the average horsepower has shot from 170 to 230. A 300 HP car used to be a rare performance beast, but they are now extremely common in the luxury/sport segment.

You can see it very nicely in this graph of Horsepower vs. MPG over time . As much as I appreciate this, it's pretty retarded and obviously way out of step with the times. The auto industry seems really really slow to react. People have wanted more economy for 5 years now, but they have just kept churning up horsepower. Now that gas prices are falling back down, the industry is just about to start putting out more efficient cars.

Anyway, what this means is there are lots of ordinary cheap cars that are pretty damn good. For example, the Mazda 6 which is one of the most reliable cars you can buy, costs around $25k and has a 3.7L V6, 24 valves, 272 hp @ 6250 rpm. Not bad !

Another new cheap asian sporty car coming out is the Hyundai Genesis. I know a lot of people have bad associations with "Hyundai", but if you take the sensible position that "build quality" means it won't fall apart and break down when you drive it, then a Hyundai is actually a better built car than any of the supposedly well engineered German cars.

The Genesis has 306 hp and 266 lb-ft of torque from a 3.8-liter V6. It's RWD and is basically a direct compentitor to the Infinity G37 Coupe. But it costs $30k , vs. $40k for the Infinity (and around $50k for a comparable BMW). And actually the Genesis has much more low-end torque than either the G37 or a BMW 3, both of which need high RPMs to make power. (IMO a flat torque curve is much much better).

I also really like the styling of the Genesis. While the Germans seem to be racing each other towards rice-boy baroque over-decoration with unnecessary bangles and slashes, the Genesis has a sweet smooth simple styling, that's maybe a tad boring, but at this point the only choices in car styling is to pick the one that is least bad.

Check out the direct comparison of Hyundai Genesis vs. Infinity G37 .

Sadly the Genesis is only RWD, and while the G37 comes in AWD, only the automatic is available, and the manual is only RWD. It reminds me of the dumb Lexus models that only offer their better engine with automatics. WTF. I think a manual is crucial. Though a no-stall first gear would be nice for rush hour traffic.

It would be really useful to be able to see a history of eBay final sale prices. You can search for "Nissan GT-R" on eBay, but it's hard to tell from the current prices what the actual sale prices are. I'd like to see a chart of sale prices over time like NexTag does. I guess they intentionally hide that info, but it would be pretty easy to scrape if you had a bunch of spiders and a fat pipe.

Also see consumer reports reliability summary and warranty reports by manufacturer.

BTW just looking at the ranking order of warranty reports is a bit deceptive. There's really a huge step :

Group A : < 10% very good

# 1. Mazda - 8.04%
# 2. Honda - 8.90%

(! paradigm shift here !)
Group B : not bad

# 3. Toyota (*) - 15.78%
# 4. Mitsubishi - 17.04%
# 5. Kia - 17.39%
# 6. Subaru - 18.46%
# 7. Nissan - 18.86%
# 8. Lexus - 20.05%

Group C : not good , around 25%

# 9. Mini - 21.90%
# 10. Citroen - 25.98%
# 11. Daewoo - 26.30%
# 12. Hyundai - 26.36%
# 13. Peugeot - 26.59%
# 14. Ford - 26.76%
# 15. Suzuki - 27.20%
# 16. Porsche - 27.48%
# 17. Fiat - 28.49%
# 18. BMW - 28.64%

Group D : very bad - 30% +

# 19. Vauxhall - 28.77%
# 20. Mercedes-Benz - 29.90%
# 21. Rover - 30.12%
# 22. Volvo - 31.28%
# 23. Volkswagen - 31.44%
# 24. Jaguar - 32.05%
# 25. Skoda - 32.12%
# 26. Chrysler - 34.90%
# 27. Audi - 36.74%
# 28. Seat - 36.87%
# 29. Renault - 36.87%
# 30. Alfa Romeo - 39.13%
# 31. Saab - 41.59%
# 32. Land Rover - 44.21%
# 33. Jeep - 46.36% 

(* on Toyota because apparently Toyotas made in Japan are very good and would be in Group A, but Toyotas made in America are shit and bring down their average).

03-14-09 | Computers get Hot

I'm disappointed with how loud my HTPC is. It's got the quietest of Scythe case fans, and the HD is perfectly silent, but I still hear the drone of the fans and it bugs me.

So today I thought I would try the simple solution - just stick in my home theater cabinet and shut the door.

First problem - it's too big. I have an Antec Fusion case so it looks like a home theater component, but it's just a bit bigger than any standard amplifier, and my cabinet is standard size. Annoying, but that's not really a big deal because -

It's going to get way too hot in there. I figured I would have to cut air holes in the back of the cabinet anyway, so I'll just cut a bigger hole so that the oversize case fits. I would up cutting just a few holes, since I had to use a steak knife to punch the entry holes and then my jig saw to cut them. (I would up breaking the steak knife blade, damn shitty stamped blades).

Now it's been running in there a few hours, and it is indeed much quieter, but it's also getting boiling hot in there even with the big air holes. Damn. What I need is a large solid box with two open ends, so that the sound is muffled but it still gets plenty of air.

My next thing to try is replacing the PSU with a lower-power quiet one. They even make fanless PSU's now though that's a bit scary because it just means they pump more heat into the case.

My other option is to throw this damn thing out the window, burn down my apartment, ride my bicycle around the world and never touch another damn piece of electronics again.

03-14-09 | Merge

I've created this fucking nightmare for myself where I have like 4 different slightly different versions of the same code. I've got my home Galaxy3, my home cblib, my work Galaxy3, my work cblib, the old Exoddus code, and the Oodle Core, and they're all quite similar but a bit different.

Once a week or so I've been trying to merge the bug fixes I do in one bit to the other bits, and it's been just awful. Today on this miserable Saturday I've got a miserable big bit of merging to do. So I went and got the free trial of Araxis Merge.

On the plus side, the GUI is slightly better than P4 merge. On the minus side, the actual merge is completely miserable.

WTF. It seems completely unable to find identical blocks that are just moved.

Like if I in one file I have




And in the next file I have




it totally freaks and just shows that as a big diff. It should show me that stuff is just swapped in location.

I usually wind up doing the merge just by doing windiff between the dirs and then manually applying the edits. Not fun.

The other thing I'd like to see is detection of synonyms. A common thing I have is "gAssert" in Galaxy is "ASSERT" in cblib and "RR_ASSERT" in Oodle. You should be able to detect that. Even if it was a manual config file of synonyms associated with directories that would be okay.

03-13-09 | Automatic Prefs

I wrote briefly at molly about the way I do "hot vars" or "tweak vars". I don't really like them. I had a bunch in my GDC app for tweaking, but it means I have to edit the code to tweak things and I don't like that for various reasons. It's way better to have a real text pref file. There are just so many wins. I can source control it seperately. I can copy it off and save good snapshots of prefs I like for different purposes, like a development prefs vs. final run prefs. I can copy it and change it to make new instances. And I can edit it and have my final app load the changes without a recompile (hot var can do this too if you keep the C file around as "data")

Fortunately I have a prefs system in cblib, so I switched to that. But it made me realize that I'd really like to have an automatic pref system. Basically I want to write something like :

struct MyPref

int i = 7;
float x = 1.3;
String str = "hello world";
ColorDW color(200,50,177);
Vec3 v(3.4,0,1.7);


and have it automatically get IO to the pref file and construction with those values as defaults.

Now, that all is actually pretty easy if I just make some custom syntax and run a source code preprocess over the file before compiling. I could use a syntax like :

struct MyPref

int i; //$ = 7;
float x; //$ = 1.3;
String str; //$ = "hello world";
ColorDW color; //$ (200,50,177);
Vec3 v; //$ (3.4,0,1.7);


where anything with a //$ automatically becomes a "pref var" that gets IO'd and tweakability and so on.

The annoyance and something I haven't figured out is just how to deal with generated code in a build setting. Should the code generator stick the generated code back into the original file? Should it make another seperate C file on the side and put the generated code in there? Maybe all the generated code from all the C files in the whole project should go together in one big spot?

I dunno but it seems like a mess. Maybe the easiest thing to do would be to put the autogenerated code in the same file, and run the generator as a pre-build step.

It would also be annoying to have to put the code generator in as a pre-build step on every file one by one. So annoying as to be unacceptable actually. I would want it to automatically run on all my files any time I build, so that if I just go and put the autogen markup in any file it does the stuff and I don't have to open msdev options dialogs.

I know people out there are doing things like this, so I'm curious how you deal with the mod time issues and builds and source control.

BTW the autogenerated code will look something like this :

void MyPref::AutoGen_SetDefault()
    i = int(7);
    x = float(1.3);
    str = String("hello world");
    color =  ColorDW(200,50,177);
    v = Vec3(3.4,0,1.7);

template <typename functor>
void MyPref::AutoGen_Reflection()

pretty easy to generate just by some text movement. The big win of the shortened syntax is that you only have to write a variable once instead of 3 times.

03-13-09 | Paged Pool Leaks

So I tracked down the paged pool leak. It appears to be a bug in my ATI driver (or perhaps a bug in the DX interface that shows up as a leak in the driver). It's caused by locking POOL_MANAGED textures that are currently in the GPU push buffer. When you do that it causes the hardware texture to get aliased to avoid contention, and it appears something in there is leaking. I don't think that whole textures are leaking because the leak is pretty slow, it must just be some kind of texture book-keeping object (eg. maybe it actually is the "texture" struct, just not the actual surface bits).

Anyhoo, nobody cares about bugs that I see via Directx8, but what is interesting is the cool Poolmon utility.

Poolmon lets you see the kernel allocations and thus track down leaks.

You need to turn on some registry settings .

Run Poolmon in a command line with lots of vertical lines.

Use the keys "d" and "p" to get the view you want

Then track away.

03-13-09 | Windows Game Development

... blows so bad. Can't we just all be Xenon-exclusive? Please?

In a bit more detail - I've been tracking down a graphics driver bug the last few days. And this is like the easy case - it's on Windows XP and it's happening on my dev machine. I remember the old days of hell at Eclipse/Wild Tangent where we'd get a report that something in the graphics screws up, but only on Windows version XX and only with graphics card YY and only with driver version ZZ and only when you run Media Player at the same time, and we can't repro it any other way.

At WT we had a huge room full of like 50 computers with all kinds of different hardware and OS setups. Any time we did a release it had to be tested on all those boxes. (this was back in the day when you had 3d-only cards like the Voodoo and NVidia had just come out with the Riva 128, and you had to support all kinds of weird ass cards, in comparison the differences between even the most different of cards these days are very minor). I'm actually amazed how many PC game development studios don't seem to have this kind of test room.

We also had 2 full-time employees basically running testing all the time. I'm amazed at how many game studios have zero internal testing.

03-12-09 | Fixed Memory Pools

The Windows Kernel Paged Pool shit I'm dealing with is reminding me how much I hate fixed size memory pools.

I have fucking 3 Gigs of RAM in this machine and I'm using less than 1 G ! How can you be running out of memory !!!! Oh, because you have a fucking stupid fixed size page thing. Now, okay, maybe *MAYBE* the OS kernel is one case in which fixed size pools is a good thing.

It's common wisdom in game development that dynamic allocations in games are bad and that "mature" people use fixed size pools because it's more stable and safe from fragmentation and robust and so on. Hog wash!

It should be obvious why you would want variable memory allocation. Memory is one of our primary limiting factors these days, and it should be allocated to whatever needs it right now. When you have fixed pools it means that you are preventing the most important thing from getting memory in some case.

For example, your artists want to make a super high poly detailed background portion of the game with no NPC's. Oh, no, sorry, you can't do that, we're reserving that memory for 32 NPC's all the time even though you have none here. In another part of the game, the artists want to have super simple everything and then 64 NPC's. Oh no, sorry, you only get 32 even though you could run more because we're reserving space for lots of other junk that isn't in this part of the game.

Now, I'm not saying that budgets for artists is a bad thing. Obviously artists need clear guidelines about what will run fast enough and fit in memory. But having global fixed limits is a weak cop out way to do that.

Furthermore, recycling pools and maximum counts for spawned items is a perfectly reasonable thing to do. But I don't think of that as a way of dividing up the available memory - it's a way of preventing buggy art from screwing up the system, or just lazy artists from making mistakes. For example, having a maximum particle count doesn't mean you should go ahead and preallocate all those particles, cuz you might want to use that memory for something else in other cases (and of course the hard-fixed-size pool thing can

In general I'm not talking here about *dynamic* variation. I'm talking about *static* variation. Like anything that can be spawned or trigger from scripts or whatever, stuff that can be created by the player - that stuff should be premade. Anything that *could* exist at a given spot *should* exist. That way you know that no player action can crash you. Note that this really just a way to avoid testing all the combinatorics of different play possibilities.

By static variation I mean, in room 1 you might have resource allocation like {16 NPC's, 100 MB of textures} , in room 2 you might have {8 NPC's, 150 MB of textures}.

Fixed sized budgets is like if you partitioned your hard disk in half for programs and data. People used to do things like that, but we all now realize it's dumb, it's better just to have one big disk and that way you can change how you are using things as need arises.

Now, people sometimes worry about fragmentation. That may or may not be an issue for you. On Stranger on XBox it basically wasn't an issue because we had 64M or physical memory and 2G of virtual address space, so you have tons of slack. Again now with 64 bit pointers you have the same kind of safety and don't have to worry. Sadly, 32-bit Windows right now is actually in a really bad spot where the amount of physical memory roughly matches the address space, and we actually want to use most of that. That is fragmentation danger land.

However, doing variable size budgets doesn't necessarily increase your fragmentation at all. The only thing that would give you fragmentation is if you are dynamically allocating and freeing things of different sizes. Now of course you shouldn't do that !

One option is just to tear things all the way down and build them all the way back up for each level. That way you allocate {A,B,C,D} in order, then you free it all so you get back to empty {} , then next level you allocate {C,B,B,A,E} and there's no fragmentation worry. (if you tried to do a minimal transition between those sets by doing like -D +B+E then you could have problems).

Another option is relocatable memory. We did this for Stranger for the "Contiguous" physical memory. Even though virtual address fragmentation wasn't an issue, physical memory (for graphics) fragmentation was. But all our big resources were relocatable anyway because they were designed for paging. So when the contiguous memory got fragmented we just slid down the blogs to defrag it, just like you defrag a disk. Our resources were well designed for fast paging, so this was very fast - only a few thousand clocks to defrag the memory, and it only had to be done at paging area transitions.

Note that "relocatable resources" is kind of a cool handy thing to have in any case. It lets you load them "flat" into memory and then just rebase the whole thing and boom it's ready to use.

Personally after being a console dev and now seeing the shit I'm seeing with Oodle, I would be terrified of releasing a game on a PC. Even if you are very good about your memory use, your textures and VB's and so on create driver resources and you have no idea how big those are, and it will vary from system to system. The AGP aperture and the Video-RAM shadow eat out huge pieces of your virtual address space. The kernel has an unknown amount of available mem, and of course who knows what other apps are running (if it's a typical consumer machine it probably has antivirus and the whole MS bloatware installed and running all the time).

I don't see how you can use even 512 MB on a PC and get reliable execution. I guess the only robust solution is to be super scalable and not count on anything. Assume that mallocs or any system call can fail at any time, and downgrade your functionality to cope.

Now, certainly doing prereserved buckets and zero allocations and all that does have its merits, mainly in convenience. It's very easy as a developer to verify that what you're doing fits the "rules" - you just look at your allocation count and if it's not zero that's a bug.

It's just very frustrating when someone is telling you "out of memory" when you're sitting there staring at 1 GB of free memory. WTF, I have memory for you right here, please use it.

The other important thing is efficiency is irrelevant if you can't target it where you need it. Having a really efficient banana picking operation doesn't do you a lick of good when everyone wants apples. A lot of game coders miss the importance of this point. It's better to run at 90% efficiency or so, but be flexible enough to target your power at exactly what's needed at any moment.

Like with a fixed system maybe you can handle 100 MB of textures and 50 MB of geometry very efficiently. That's awesome if that's what the scene really needs. But usually that is not the exactly ideal use of memory for a given spot. Maybe some spot really only needs 10 MB of geometry data. You still only provide 100 MB of textures. I'm variable and can now provide 139 MB of textures. (I lose 1 MB due to overhead from being variable).

Many game devs see that and think "ZOMG you have 1 MB of overhead that's unacceptable". In reality, you have 39 MB less of the actual resource your artists want in that spot.


ERROR_NO_SYSTEM_RESOURCES is the fucking devil.

So far as I can tell, the only people in the history of the universe who have actually pushed the Windows IO system really hard are me and SQL Server. When I go searching around the web for my problems they are always in relation to SQL Server.

I don't have a great understanding of this problem yet. Hopefully someone will chime in with a better link. This is what I have found so far :

You receive error 1450 ERROR_NO_SYSTEM_RESOURCES when you try to create a very large file in Windows XP
SystemPages Core Services
Sysinternals Forums - not enough resources problem - Page 1
Overlapped WriteFile fails with code 1450 [Archive] - CodeGuru Forums
Novell Eclipse FTK file io
How to use the userva switch with the 3GB switch to tune the User-mode space to a value between 2 GB and 3 GB
GDI Usage - Bear
Download details Detection, Analysis, and Corrective Actions for Low Page Table Entry Issues
Counter of the Week Symptoms Lack of Free System Page Table Entries (PTEs) and Error Message ERROR_NO_SYSTEM_RESOURCES (1450
Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003

Basically the problem looks like this :

Windows Kernel has a bunch of internal fixed-size buffers. It has fixed-size (or small max-size) buffers for Handles, for the "Paged Pool" and "Non-Paged Pool", oh and for PTEs (page table entries). You can cause these resources to run out at any time and then you start getting weird errors. The exact limit is unknowable, because they are affected by what other processes are running, and also by registry settings and boot.ini settings.

I could make the error go away by playing with those settings to give myself more of a given resource, but of course you can't expect consumers to do that, so you have to work flawlessly in a variety of circumstances.

In terms of File IO, this can hit you in a whole variety of crazy ways :

1. There's a limit on the number of file handles. When you try to open a file you can get an out-of-resources error.

2. There's a limit on the number of Async ops pending, because the Kernel needs to allocate some internal resources and can fail.

3. There's a limit on how many pages of disk cache you can get. Because windows secretly runs everything you do through the cache (note that this is even true to some extent if you use FILE_FLAG_NO_BUFFERING - there are a lot of subtleties to when you actually get to do direct IO which I have written about before), any IO op can fail because windows couldn't allocate a page to the disk cache (even though you already have memory allocated in user space for the buffer).

4. Even ignoring the disk cache issue, windows has to mirror your memory buffer for the IO into kernel address space. I guess this is because the disk drivers talk to kernel memory so you user virtual address has to be moved to kernel for the disk to fill it. This can fail if the kernel can't find a block of kernel address space.

5. When you are sure that you are doing none of the above, you can still run into some other mysterious shit about the kernel failing to allocate internal pages for its own book-keeping of IOs. This is error 1450 (0x5AA) , ERROR_NO_SYSTEM_RESOURCES.

The errors you may see are :

ERROR_NOT_ENOUGH_MEMORY = too many AsyncIO 's pending
Solution : wait until some finish and try again

ERROR_NOT_ENOUGH_QUOTA = single IO call too large
Solution : break large IOs into many smaller ones (but then beware the above)

ERROR_NO_SYSTEM_RESOURCES = failure to alloc pages in the kernel address space for the IO
Solution : ???

So I have made sure I don't have too many handles open. I have made sure I don't have too many IO ops pending. I have made sure my IO ops are not too big. I have done all that, and I still randomly get ERROR_NO_SYSTEM_RESOURCES depending on what else is happening on my machine. I sort of have a solution, which seems to be the standard hack solution around the net - just sleep for a few millis and try the IO again. Eventually it magically clears up and works.

BTW while searching for this problem I found this code snippet : Novell Eclipse FTK file io . It's quite good. It's got a lot of the little IO magic that I've only recently learned, such as using "SetFileValidData" when extending files for async writes, and it also has a retry loop for ERROR_NO_SYSTEM_RESOURCES.

Further investigation reveals that this problem is not caused by me at all - the kernel is just out of paged pool. If I do a very small IO (64k or less) or if I do non-overlapped IO, or if I just wait and retry later, I can get the IO to go through. Oh, and if you use no buffering, that also succeeds.

03-12-09 | Bad Chef, Bad

There's a new trend in fancy restaurants of putting your food in a pool of water rather than a proper sauce. This is fucking retarded and should stop immediately.