Leveraging Structure and Context for Language-Adjacent Representation Learning