I was just in San Diego after Thanksgiving this year!
So I can tell you what we DID do with a bit over 2.5 days in San Diego...although I was with my mom so it wasn't exactly romantic.
Friday night - Seaport Village - if you like shopping it's OK and it IS by the water, which is nice. The counter-service food is mediocre and very expensive, but the sit-down restaurants may be nicer. VERY tourist-y.
Saturday morning - Old Town San Diego - touristy but has historical exhibits - I could've spent several hours here, I think. We got Mexican food at one of the restaurants and it was OK.
Saturday afternoon - San Diego Maritime Museum - if you or your DH like tall ships/pirates/sailboats/Russian submarines/the film "Master and Commander", then I'd consider it a must-see. All the exhibits are actually ON the ships themselves, so if your mobility is limited it might be a challenge. We took a 2 (?)-hour cruise on a real sailing ship, which was fun, but I'd probably skip that since you're doing a harbor cruise anyway. (If you are the type who likes to get involved and ask questions, the sailing cruise would be more fun.) If you go and there's not a special event going on, be sure to go up to the second floor of the ferry/gift shop area, where the bathrooms happen to be. The area up there is so pretty! They use it for wedding receptions and such.
Saturday night - Horton Plaza - food court was much better than Seaport Village's. I got the most delicious food (and a lot of it!) from the Gyro place for less than $6. Mom bought a pretty cupcake from a fancy cupcake place.
Sunday - Old Town Trolley Tours - I didn't like it at all, as our first guide was incredibly obnoxious, and since it was during the winter, the hours were short and we could only get off at one stop. We took it to Coronado Island...I would have liked to look around the Hotel Del Coronado but we didn't have time. Lots of shopping, though--my mom loved a place called "The Papery"--and one of the Coronado stops is right by a pub...some of the guests took great advantage of that.
Monday - Belmont Park - you can see the beach and ride a roller coaster from 1925! (Pricey but fun--a lot more exciting than I'd expected.) They had a cute table service beachside restaurant with OK prices. It was cold and a Monday morning when we went so there weren't any crowds at all.
We also used the REAL trolley (light rail) system--there's a station right at Old Town, and when we went it was $5 for a day pass. It can take you to or near several of the tourist spots if you don't feel like driving; we took it from Old Town to the Maritime Museum and then back up to Horton Plaza. (I preferred it to the expensive Old Town Trolley.) But then, I don't think "mass transit" goes along with "romantic getaway".
Oh, and if I were going, I would have to drive to Jack in the Box headquarters to see if fans/employees are really leaving cards and teddy bears and stuff at Jack's parking space.
Have a great trip!