Modeling Affect in Speech and Language in the Presence of Natural Inconsistency