Breast cancers with neuroendocrine (NE) differentiation are very heterogeneous, comprising broadly cancers that are morphologically similar to NE tumors (NET) of other anatomic sites, infiltrating breast carcinomas, no special type (IBC-NST) and other special subtypes with NE morphology and/or NE markers expression. Depending on the classification schemes, they are variably included into "NE breast cancers". The latest WHO classification harmonized NE breast cancers with NE neoplasms (NEN) of other organ systems, defined NEN into well-differentiated NET (low Nottingham grade) and poorly-differentiated NE carcinoma (NEC) (high Nottingham grade). Other IBC with NE differentiation are diagnosed based on solely the non-NEN component. Due to the changes in diagnostic criteria, variable results were obtained in the previous studies on NE breast cancers. Hence, the clinical value of NE differentiation in breast cancers is not well investigated and understood. In this review, the current understanding in the pathogenesis, clinical, prognostic, immunhistochemical, and molecular features of "NE breast cancers" is summarized. Controversial issues in their diagnosis and classification are also discussed.