Multiple choice questions (MCQs) and short essay questions (SEQs) are common methods of the assessment of medical students in courses of pharmacology. Poorly constructed test items (questions) are a widespread problem resulting in failure to assess learning objectives. It has been reported that there are 36.0% to 65.0% flawed test items in medical education assessment tools. Thus, the objective of this study was to evaluate MCQs by determining the item writing flaws (IWFs) and to evaluate the SEQs by determining the cognitive level of each item. Four pharmacology tests were administered to third^-year pharmacy students at Department of Pharmacology, Faculty of Pharmacy, Omar Al-Mukhtar University, Bayda, Libya. These were evaluated by determining the IWFs and the level of the cognitive domains. Based on Buckwalter’s modification of Bloom’s taxonomy cognitive level, for the SEQs, 30.0% of the questions were attempted to check recall of information, 26.0% were attempted to evaluate understanding and interpretation of data and 43.0% of the questions were attempted to check the application of knowledge for solving a particular problem. For the MCQs, 94.6% of the questions were attempted to evaluate the understanding and interpretation of data. For the IWFs, there were more than 40.0% of flawed questions. The most common writing flaws were the negative stem (47.4%), unfocused item (16.0%), non-homogenous in grammar and contents (10.0%), all the above (10.0%) and clang association (05.0%). In a short essay, the SEQs were of excellent quality because they were equally distributed among the three levels of cognitive (level I, II and III). On the other hand, the most common mistakes IWFs of the MCQs were the negative stem (47.0%) and the idea was not clearly and concisely stated in the stem (16.0%). This study concludes that questions in SEQs are valid to measure the learning objective but MCQs were not in pharmacology courses in Libya.