The Victoria’s appeal has as much to do with the quaintness of Oxshott, the charming staff and serendipitous weather, as it does the food, writes Hannah Twiggs