Oldswinford Castle, never a true castle, was converted to apartments many years ago