This article has for foundation a textual dialogue between Baldwin and Freud. It argues that the imprint left by nineteenth and early twentieth-century racial metaphors on the Freudian construction of gender and sexuality has reproduced the logic of racial differentiation within psychoanalysis. This can be seen, for instance, in the mutual exclusion of identification and desire and the role played by unconscious fantasies. Baldwin’s exploration of American white consciousness provides the lens for perceiving this racial undercurrent in Freud’s texts, and more particularly in his 1915 sketch of the fantasy. In this text, Freud underlines the double nature of the fantasy which partakes at the same time of the unconscious and the preconscious. Meanwhile, he also draws an analogy with the so-called “half-breed.” As it is mirrored by the supposed racial duality of the mixed race individuals, the ambivalence of the psychic location of the fantasy will happen to bear some importance in the Freudian account of gender and sexual identities. Through the mediation of the fantasy, the relation between race and gender and sexualities evolves in a complex interplay of identity and difference best exemplified by the figures of speech of the analogy and the metaphor, both aiming at scientific and objective truth. The possible counter-truth which has vanished in the process is that of the “half-breed” whose voice is being silenced. It is through the writing of consciousness and intimacy of James Baldwin that this silenced voice can be heard again as he strives to dismantle the cultural logic of differentiation and the lethal power of metaphorization and stifling fantasies.