Using Generative AI to Build Complex Free-Response Questions

Overview

Students need sustained opportunities for practice so they can apply new knowledge and skills and address the feedback they receive from practice assignments to improve their learning. Providing students multiple opportunities for practice is also important to allow them to build familiarity with different question formats found on standardized tests and state tests to build their confidence for the actual assessments. Examples include questions from standardized exams like document-based questions on Advanced Placement (AP) exams and specimen papers on the International Baccalaureate (IB) exams, in addition to teacher-created formats like essay prompts and analysis questions.

However, developing rich and complex free-response questions aligned to these assessments takes significant time, and organizations that oversee these standardized tests only release a limited amount of sample questions accessible to educators. Generative AI tools can help teachers develop these questions more quickly, giving students more opportunities to apply their learnings and receive feedback to strengthen their knowledge and skills. 

Example From the School Teams AI Collaborative

Zach Kennelly, a Civics and AP Psychology teacher at DSST: College View in Denver and a participant in the School Teams AI Collaborative, wanted to make sure his AP Psychology students had multiple attempts at the AP exam’s Article Analysis Questions (AAQ). The AAQ gives students a summary of a peer-reviewed study and involves multiple elements that ask students to identify the research design of the study, analyze statistics, convey the results of the study, and more. Given the complexity of these questions, Kennelly estimated that students need to complete about 35 AAQs to be prepared for the AP exam.

Developing exam-aligned AAQs is time-intensive. The College Board, which oversees AP exams, does release AAQs from previous exams – but the question bank is limited, and changes in the AP Psychology exam signify that previous questions are not fully aligned to the revised course. Kennelly estimated that he spent anywhere from 30-75 minutes to develop a single AAQ, depending on the content. Using generative AI to support the process decreased this time to less than 15 minutes, allowing him to create five AAQs per unit (compared to his previous output of one per unit). Kennelly was able to achieve this through the following steps:

• Drafting Effective Prompts: Kennelly used Playlab, an AI tool that enables educators to build their own AI-enabled applications, to develop an AAQ-generating chatbot. He configured the chatbot’s prompt as follows: “You are an AP Psychology expert and assessment designer. Your task is to craft an adapted research study based on an actual published psychology article and create a rigorous, College Board-style Article Analysis Question (AAQ) that assesses students’ ability to analyze psychological research.” While this required a substantial upfront investment of time, it ultimately proved efficient. Now, he only needs to enter the desired unit and topic, and the chatbot automatically generates a corresponding AAQ, complete with a summary of a peer-reviewed study, related questions, and a scoring rubric.

• Uploading Exemplars: Kennelly uploaded sample AAQs as reference documents and set guidelines (e.g., require complete sentence responses using precise psychological terminology) to ensure the chatbot would mimic their structure and language.

• Reviewing and Modifying Outputs: Once the chatbot generated the AAQ, Kennelly reviewed its output and made any modifications to make it read more like a question from an actual AP exam. He also engaged in a quick Google search to verify that the study was real, noting that extra precaution was necessary as generative AI may occasionally “hallucinate” and make up information.

Kennelly commented that creating a chatbot is not necessary for all teachers but noted anyone is welcome to use his AAQ generator. 

Apply This Strategy in Your Context

Below are ways educators can use generative AI tools to create complex free-response questions in their own context:

1. Create a Clear Prompt: Get clear and specific on what you would like the tool to produce. Use this as a sample prompt: “You are a [subject, grade level you teach] that is skilled in developing writing assignments for your students. Your task is to generate a [type of free response question] that asks students to [what is the objective of the writing assignment].”

   • Consider uploading sample assignments (publicly available exam questions from previous AP exams) to create outputs that are strongly aligned to the actual test.

   • Include in the task if you want the output to include specific types of sources (e.g., a poem, historical documents).

   • Sample prompt: “You are a ninth grade English Language Arts teacher that is skilled in developing writing assignments for your students. Your task is to generate a compare-and-contrast paper prompt for your students that asks them to explore the use of figurative language in two poems written by two 20th-century female poets. Include the text of the two poems and an exemplary response.”

2. Review and Edit What the AI Tool Creates: Review the output and edit it to tailor the content to your students and instruction. If you ask the tool to pull from an external source, verify that the source is accurate and truly exists. For example, in the sample prompt above, perform a web search on the poets identified and verify that they are female and from the 20th century. Additionally, look up the poems themselves and verify that the text is accurate.

By leveraging generative AI tools, teachers can save valuable time while providing students with more opportunities to practice and receive feedback on complex, free-response questions. As educators like Zach Kennelly have demonstrated, AI can streamline the creation of high-quality assessments, allowing teachers to focus on what matters most — supporting student learning and growth.

This AI-enabled strategy was developed by a member of the School Teams AI Collaborative — a partnership between Leading Educators and The Learning Accelerator (TLA). The Collaborative was developed to bring together innovative educators from schools across the country to share ideas and discover effective ways to use AI in the classroom.