True. I also imagine that a secret passage would have entrances designed to close automatically after a very short period of time to help conceal them, otherwise an absent-minded Deryni might accidentally leave one open, inviting passers-by to explore the hitherto unsuspected passageways. Which, of course, would be a rather glaring security breach in a castle. You really wouldn't want just anyone able to walk into Kelson's apartment....
But yeah, it still begs the question, what if someone were to find a way in, but couldn't figure out how to get back out?