Karl sets out for the road to Oxford. The inn keeper's son accompanies him to show him the way. It's quite clear that the lad took Karl by a footpath across the Old Deer Park, the gardens of George III's palace at Kew (now Kew Gardens botanical research establishment) on the right, as Karl mentions.
So, we can safely say that Karl Moritz left Richmond via the ferry across to Isleworth. Karl writes about the houses he could see across the river. I think we can be absolutely certain that he was referring to the brick mansion, if not indeed also the pinkish looking "temple shaped" house as well.
This is therefore almost exactly the view he saw as he approached the steps for the Isleworth ferry, and wrote about in his letter.