Galaxy formation through the lens of galaxy structure with semi-empirical models and deep learning