I've been happy with the 35 f/2. It's become the primary lens that I keep on my Canon 20D, although the 17-40 f/4L spends a fair bit of time there too.
I'm enjoying the 35 because it's so small. In the nearly three years I had it, I never shot my D30 without the vertical battery grip. I haven't been able to get the new BG-E2 grip for my 20D yet. At first, the 20D felt incredibly tiny and uncomfortable. I still have problem getting portrait-orientation pictures straight without the grip but I love the portability.
I just need to get one of the Op/Tech Pro Loop straps and I'll truly be set to have the 20D with me at all times.
Back to the lens: I like the "normal" field of view it offers and especially like being able to open up wider than f/4.
Robert...I find the 35mm a bit limiting...for travel purposes I would want a wider lens. I think the 17-40 would probably be the best for me if I had to take only one lens.
Isabel