Background: The science of medical education recommends validation of Multiple Choice Questions (MCQs) framed by teachers by a procedure called objective item analysis. Many medical college teachers however do not adopt this procedure and use their subjective judgment and expertise to validate MCQs. Wide variation between the subjective and objective analysis of MCQs may reduce the validity of such an assessment based on subjectively validated MCQs. Aims and objectives: This study was undertaken to assess the degree of agreement between subjective and objective MCQ validation with regards to three parameters - difficulty index, discrimination index and distracter effectiveness. Materials and methods: Five teachers from different departments were explained about these parameters and the cut off values used to categorize MCQs into groups based on these parameters. These teachers categorized 20 MCQs and 60 distracters each (total 100 MCQs and 150 distracters) into groups using their subjective judgment and expertise followed by an objective categorization of the same MCQs by the authors using the objective item analysis procedure. The degree of agreement between subjective and objective item analysis was expressed as sensitivity value of the subjective item analysis with objective item analysis as the gold standard. Results: Data analysis revealed that there was wide variation in sensitivity values between teachers and maximum sensitivity value reached was 70% for all three parameters. Conclusion: There is variable and relatively low sensitivity of subjective item analysis procedure and hence the objective analysis procedure must be used to determine the difficulty index, discrimination index and distracter effectiveness of MCQs.