Despite the proliferation of Artificial Intelligence (AI) technologies, their uptake in clinical settings has been lacking progress due to complexities of sociotechnical factors and intricacies of decision-making. Fairness and bias of predictive models, ethics and quality of training data, and corresponding compliance requirements become especially pressing while remaining fuzzy and implicit for various stakeholders who make the decisions. We present learnings and future directions from a design study with domain experts and propose a novel approach to encoding and collaborative reasoning on complex requirements for AI-Empowered Clinical Decision Support System (AI-CDSS) design based on Knowledge Graph (KG) representation. The insights will be useful to the community of visualization researchers who work on ethical AI-CDSS design and conduct design studies with clinical partners.