I just got back from a 3 day hike in the Budawangs including Byangee walls, the castle and monolith valley. On the the first day we left Yadboro camp ground and headed up the ridge to castle gap and on to climb Byangee mountain.
The big question we had was "can we traverse from castle gap on the Byangee mountain track to the Kaliana ridge track around the base of the castle?"
From a recent trip I thought that it should be possible so we headed off that way. Either others had done it or there was a good reason they hadn't! It turns out that there is a faint track that traverse right at the bottom of the cliff line. To find the track head west right next to the cliff where the ridge meets the castle's lower cliff line.