Try the Map class's eachLayer() method:
var layers = ;
if( layer instanceof L.TileLayer )
For complex cases you would end up writing your own layer switcher, and I don't see anything wrong with that. The Leaflet library aims to provide a minimum of core functions for an interactive map. If you need a bigger, all-in-one solution, try OpenLayers.