Evaluating Language Model Math Reasoning via Grounding in Educational Curricula (2024)

Math benchmarks for evaluating LMs.

Past work assessing mathematical reasoning capabilities of LMs have mainly focused on grade school arithmetic and algebra and problems that are easy to validate. These datasets may be written by annotators without formal pedagogical training (e.g. Cobbe etal., 2021; Patel etal., 2021; Amini etal., 2019; Ling etal., 2017; Miao etal., 2021),synthetically generated by LMs (Mitra etal., 2024; Liu etal., 2023a), and derived from advanced math competitions (Hendrycks etal., 2021).In contrast, our work characterizes real-world math curricula within an expansive, fine-grained taxonomy of math skills and concepts (§3).Concurrent work by Mishra etal. (2024) presents an approach for generating problems that align with math standards, but targets a limited set of easily validatable standards.Our work instead focuses on the task of labeling problems across 300+ comprehensive and challenging K-12 standards, and emphasizes ecological validity by working with educators (§4.2). Earlier work in education has identified math standards in problems with models such as SentenceBERT Li etal. (2024b), task-adapted BERT Shen etal. (2021), and support vector machines Karlovčec etal. (2012). Our work re-envisions this task into a format suitable for evaluating instruction-tuned LMs.

LMs for improving math education.

LMs have increasing potential for improving math education through applications like automated feedback to educators (Jacobs etal., 2022; Demszky etal., 2023; Wang and Demszky, 2023),automated tutoring (Hobert and Meyervon Wolff, 2019; Wang etal., 2024b),and lesson planning (Malik etal., 2024; Kasneci etal., 2023).Evaluating models with education domain experts before real-world deployment is crucial to best align with educational needs and mitigate potential harms (Patikorn etal., 2019; Li etal., 2024b; Wang etal., 2024a).In our setting, we work with domain experts in a manner that centers real-world educational needs (§4.2).

Evaluating LMs amid fine-grained differences.

Our emphasis on the granularity of models’ decisions relates to other work that focuses on small yet impactful differences in language. Examples include the generation of good distractors that sufficiently challenge question solvers Gao etal. (2019); Zesch and Melamud (2014), the mining of “hard negatives” to supervise models to discern positives and close negatives Robinson etal. (2021); Kalantidis etal. (2020), and minimal modifications of model inputs that change ground truth labels Gardner etal. (2020). In our case, our “hard negatives” and distractors are not artificially constructed, but instead originate from inter-standard relationships drawn by pedagogical experts.

3.1 Task Definition & Formats

In the initial phase of this work, we met with professional curriculum reviewers to better understand their process.We learned that identifying math standards in curricular materials is time-intensive, and reviewing a set of published materials can take 6-8 months to complete. This motivated us to investigate whether models can support reviewers by combing through subtle differences across standards. In collaboration with these curriculum reviewers, we identified two usage patterns in which these experts would employ language model assistance for performing curricular alignment—verifying whether a problem aligns to a given standard and tagging a problem with all aligned standards.

In the first, reviewers may wish to verify alignment, or check a problem against a single standard. For example, one may confirm publishers’ claims of alignment (§4.1), or evaluate whether generated materials follow $\mathcal{S}$ indicated in prompt instructions (§4.2). This task format is structured as a binary yes/no question (Figure1): does a problem fully align with a given standard? Taking inspiration from textual entailment(Dagan etal., 2005) and claim verification(Thorne etal., 2018), this format also allows us to investigate models’ sensitivity to narrowing differences in positive and negative standards when we perturb them in queries.

Though verification is useful for checking problem alignment with a single standard, reviewers are also often tasked with identifying all aligned standards for a given problem.This task format, which we define as tagging, involves selecting from a provided set of candidates.Prompting language models with hundreds of candidate standardsrisks hitting context limits or models getting “lost in the middle” (Liu etal., 2023b).Instead, we take advantage of a domains-clusters-standards tree structure that experts use to organize standards into a hierarchy; related standards are grouped as leaves on this tree (Figure1).We define the tagging task as follows: given a math problem, (1) start at the top of the tree and select the best domain/s $\mathcal{D}$ it teaches, (2) then traverse to each selected $\mathcal{D}$s’ subtrees, (3) repeat selection for cluster/s $\mathcal{C}$, (4) stop traversal upon reaching standards $\mathcal{S}$ at the leaves, and (5) tag the problem with selected $\mathcal{S}$.By traversing this tree, our tagging task format allows us to see whether models can make increasingly granular distinctions among adjacent concepts/skills.

Across these task formats, we experiment with three models: Mixtral-8x7B(Jiang etal., 2024), Llama-2-70b-chat(Touvron etal., 2023), and GPT-4-Turbo(Achiam etal., 2023). We unify them under a single wrapper, which handles input truncation and API calls (AppendixB).

ATC: Math Standards Dataset
Grade levels		K-12
# of grades & domains, e.g. 4.NBT		65
# of clusters, e.g. 4.NBT.A		147
# of standards, e.g. 4.NBT.A.1		385
# of connections b/t $\mathcal{S}$		1,040
Avg standard description length		36.23
MathFish: Problems Dataset
	IM	FL	Total
# of examples	19,570	2,206	21,776
# of labeled examples	7,735	2,188	9,923
Total words	4.5M	199K	4.7M
Avg problem length	230.51	90.20	216.30
Grade levels	K-12	3-11	K-12
# of unique standards	366	287	366

3.2 Math Standards & Organization

To facilitate these experiments, we also introduce two datasets: Achieve the Core (ATC) describes math standards and their organization, and MathFish links standard labels to problems (Table1). Both involve Common Core State Standards (CCSS), which offer fine-grained and comprehensive coverage of K-12 math skills/concepts Porter etal. (2011).

ATCconsists of CCSS standard labels (e.g. 4.NBT.A.1) and descriptions (e.g. Recognize that in a multi-digit whole number…) from Achieve the Core’s coherence map, which captures how pedagogical experts characterize, organize, and categorize math.²²2github.com/achievethecore/atc-coherence-mapATC includes two types of relationships among standards: as leaves in a topical hierarchy, or as a graph of conceptual connections. Our experiments on verifying (§4) and tagging (§5) use these relationships to show how related standards that differ subtly can challenge models.

As described in §3.1, the CCSS hierarchy traverses the following levels from top to bottom: grade, $\mathcal{D}$, $\mathcal{C}$, and $\mathcal{S}$.³³3In high school, domains analogous to K-8 ones are called “categories”. More details on how we simplify complexities of CCSS like this one can be found in AppendixD. In addition, ATC includes conceptual connections between $\mathcal{S}$ that cut across this hierarchy Zimba (2018). CCSS is designed to follow students’ learning progressions Team (2013), and recognizing these connections is central to enabling students to learn.That is, a problem intended to teach students addition would not jump directly to solving for variables. Figure1, under Verification, shows an example of a conceptual connection: the 4th grade $\mathcal{S}$ on the left (Recognize that in a multi-digit whole number…) progresses to the 5th grade $\mathcal{S}$ on the right.

3.3 K-12 Math Problems

We evaluate models’ abilities to identify math skills and concepts using publisher-labeled data pulled from curricular websites. Publicly accessible open educational resources (OER) provide a rich test bed for evaluating models, as they are designed to cover nearly all $\mathcal{S}$ within the grade levels they offer.We scrape pre-labeled problems from two OER that have been verified to be reputable by third party curriculum reviewers: Illustrative Mathematics and Fishtank Learning.⁴⁴4illustrativemathematics.org/math-curriculum and fishtanklearning.org/about Each problem (e.g. the Task in Figure1) in this combined dataset, which we refer to as MathFish, is a segment of these materials demarcated by $\mathcal{S}$ labels, which we map onto ATC’s natural language descriptions. A problem in MathFish can be labeled with multiple $\mathcal{S}$.Preprocessing details are in AppendixA.

We analyze models in text-only settings, replacing images with a dummy image token, and leave multimodal analyses for future work.We observed that many examples tend to include tables; we reformat HTML tables into Markdown or JSON based on models’ preferences during preliminary experiments (AppendixC). Though we contribute this entire scraped dataset of OER problems to facilitate future work, we evaluate models on a smaller evaluation set, which consists of 20% of all data. We do not evaluate models’ problem solving abilities on this data, and instead focus on identifying math skills/concepts, as not all examples contain solutions, and some “problems” are designed to be group and/or hands-on activities.

3.4 Problems’ Alignment with Standards

Alignment is not always directly evident in problems’ language, and recognizing it requires a deeper understanding of cognitive processes.In Figure1, each $\mathcal{C}$ and $\mathcal{S}$ relate to understanding the base ten number system, but differ in subtle ways. The purpose of the example problem on the left, based on the publisher’s description,⁵⁵5tasks.illustrativemathematics.org/content-standards/tasks/1562 is to teach students how an understanding of the place value system can facilitate division and multiplication by ten. If the problem were instead intended to enable students to Perform operations with multi-digit whole numbers and with decimals to hundredths, a wider range of whole numbers than powers of ten would have been included in the problem for students to fully practice those procedural skills.As another example of misalignment, the problem in Figure1 does not align with the left $\mathcal{S}$ under Verification, as the problem involves place value understanding with decimals, which are not whole numbers. Thus, small differences in mathematical language have distinct consequences for student learning and assessment.

During evaluation, we assume that all $\mathcal{S}$ not listed in MathFish do not align with a problem. We verify this assumption by having curriculum reviewers label a sample of problems paired with $\mathcal{S}$ that are not listed to align with them in publishers’ materials, but are similar to positive labels. Here, we define “similar” as $\mathcal{S}$ that are conceptually connected to positive labels in ATC’s map or are their same-$\mathcal{C}$ siblings. We recruited six teachers from a curriculum reviewing organization, which specializes in identifying CCSS alignment of materials. We paid these teachers approximately $50 an hour, and within an allotted time frame of two weeks, they reviewed 136 pairs of problems with non-listed yet similar $\mathcal{S}$. Only 3 problem and $\mathcal{S}$ pairs in this sample of assumed negatives were judged to actually be positives, thus estimating a ceiling for our assumption.

4.1 Can LMs discern (mis)aligned standards?

Experimental findings.

(i) As hypothesized, models’ accuracy decreases as negative examples become more conceptually similar to positive ones (Figure2).Regardless, the best verifier is three-shot GPT-4, scoring the highest across all positive and negative pairs.(ii) Though Hase etal. (2024) showed that grade level relates to problem solving performance, our verification task does not have this relationship, as evidenced by insignificant Spearman’s $\rho$ between grade level and F1, across all models and prompting approaches (AppendixC.5).This suggests that difficult aspects of our task extend beyond how easily a problem can be completed, posing a unique challenge for models. (iii) Finally, few-shot exemplars do not uniformly improve performance (Figure2), nor does pairing task instances with exemplars in the same or nearby grades (AppendixC.3).

4.2 Study 1: Verifying standard alignment of LM-generated problems

Diliberti etal. (2024)’s survey of teachers showed that some of the most common uses of AI in classrooms include generating materials such as assessments, lesson plans, and assignments. In addition, problem generation is a common task in AI & education research (e.g. Norberg etal., 2023; Zhou etal., 2023; Wang etal., 2021; Mishra etal., 2024), though generations are rarely evaluated by in-domain experts.In this section, we investigate the utility of GPT-4, our best verifier from §4.1, by applying it onto assessing generated math problems, working alongside K-12 math teachers to also obtain their verification judgements.

5.1 Can LMs traverse the tagging hierarchy?

Findings from assisted traversal setting.

We run several experiments that evaluate models at each level of the decision tree. Rather than have models traverse the tree on their own, we first evaluate in an “oracle”-assisted setting, where each level is presented with the assumption that the model has correctly chosen the correct branches in the previous level. We compute accuracy per branch based on the fraction of options correctly selected or not selected by the model. For example, if there are five options (A, B, C, D, E), ground truth is B, and the model selects B & E, then accuracy on this branch is 4/5.

Following our hypothesis, stronger LMs, including GPT-4 and Mixtral, tend to perform worse during assisted traversal as levels approach more fine-grained decisions (Figure3).Yet for the weaker Llama-2, few-shot examples can hurt task performance rather than help, which may be due to how some models can struggle with long in-context examples more so than others Li etal. (2024a). These model-specific findings also generalize to other prompt templates (AppendixD).Overall, the best performing model and prompt identified from these assisted traversal experiments is three-shot GPT-4 (mean accuracy = 0.850).

5.2 Study 2: Tagging problems in GSM8k

Tagging problems with math concepts/skills can help document open math datasets and benchmarks.To illustrate this, we apply our task to GSM8k. GSM8k is a popular “grade school math” dataset used to evaluate models’ problem solving abilities Cobbe etal. (2021). It has seen uptake by education-related papers Jurenka etal. (2024), as well as in reports benchmarking LMs (e.g. Touvron etal., 2023; Chowdhery etal., 2023). We expect that most problems in GSM8k should align with $\mathcal{S}$ pertaining to what Cobbe etal. (2021) describe as, “elementary calculations using basic arithmetic operations ($+$ $-$ $\times$ $\div$)”. Thus, we aim to (1) confirm that GSM8k mostly covers arithmetic and (2) relate a disaggregation of problems by skills/concepts to models’ problem solving strengths and weaknesses.

Experimental setup.

We apply our best performing tagging setup from §5 to GSM8k’s test set to approximate what $\mathcal{S}$ it may cover. For math problem solving, we experiment with a range of model sizes within three families: GPT-4, GPT-3.5; Mixtral-8x7b, Mixtral-8x22b; Llama-2-7b, Llama-2-13b, and Llama-2-70b. We run these models on GSM8k’s test set using default settings in Eleuther AI’s evaluation harness Gao etal. (2023).

Multimodality.

Math content is inherently multimodal (e.g. Lu etal., 2023), but our tasks focus on text-only data. The problems we gathered not only contain images, but sometimes embed interactive web applets. We plan to include images attached to problems in the final released form of our dataset to facilitate future work. The metadata attached to each example also includes the problem’s original url, in case others want to leverage other forms of information present in these online materials.

Curriculum review.

Throughout the paper, we engage with teachers who professionally review curricular materials. During our process of working with them, we learned the ways in which their annotations for our paper may gloss over the complexities of how curriculum review occurs in practice. For example, when we generate problems based on standards, we include only one standard in the prompt. In reality, teachers may combine multiple standards in a single lesson, or use multiple problems to collectively target one standard. In addition, we asked teachers to individually annotate problem and standard pairs, and their annotations only consist of one pass over these materials (§3.3, §4.2). During actual reviews, teachers may collect evidence of standards (mis)alignment individually, but later come together for careful deliberation, as flaws observed by one teacher may be missed by another. Finally, curriculum review typically evaluates materials for additional measures of quality beyond CCSS alignment, some of which we briefly discuss in AppendixE. We encourage future work to build on these investigations, especially as LMs become increasingly integrated into classrooms and educational technologies Diliberti etal. (2024).

A.1 Scraping MathFish

A.2 Label Preprocessing

C.1 Table Formatting

Web-scraped math problems sometimes include tables. We first experimented with different table formatting styles in one fixed prompt template: HTML, JSON, Markdown, and reStructuredText. We evaluate on a random sample of 500 verification instances, which consist of OER problems from our evaluation set paired with positive labels or negative labels sampled from $\mathcal{D^{\prime}}\mathcal{G^{\prime}}$. Using a fixed prompt template, we find that Llama-2 and GPT-4 prefer Markdown tables, while Mixtral prefers JSON (Table5). We retain these table preferences for all further experiments in our paper, including those for tagging.

C.2 Prompts

Three authors collaboratively wrote a pool of 15 prompt templates $\{\mathcal{P}_{i}\mid i=1,2,3,\ldots,15\}$ which vary in phrasing. These templates are designed to emphasize whether problems teach or enable students to learn a given concept or skill. Their paraphrases were informed by language occurring in resources that discuss Common Core alignment, especially “full intent” or “focused” alignment (Figures 10-15).¹⁰¹⁰10https://curriculum.illustrativemathematics.org/MS/teachers/design_principles.html, https://achievethecore.org/page/1118/coherence-map We again evaluate on a random sample of 500 examples as we did in §C.1.

Models vary in performance across $\mathcal{P}_{i}$ during these small preliminary experiments, though there is some overlap across models’ top three. Across all 15 prompt templates, we observe F1 ranges of 0.605-0.838 for Llama 2, 0.778-0.895 for Mixtral, and 0.778-0.913 for GPT-4. Each models’ top-3 prompts are: $\mathcal{P}_{4}$ (Figure 11), $\mathcal{P}_{10}$ (Figure 14), and $\mathcal{P}_{15}$ (Figure 15) for Llama 2, $\mathcal{P}_{1}$ (Figure 10), $\mathcal{P}_{5}$ (Figure 12), $\mathcal{P}_{15}$ (Figure 15) for Mixtral, and $\mathcal{P}_{1}$ (Figure 10), $\mathcal{P}_{4}$ (Figure 11), $\mathcal{P}_{7}$ (Figure 13) for GPT-4.

Then, we ran these top-3 performing prompt templates on the full evaluation set, across all five types of negative labels as described in the main text §4. Figure5 shows the variation we obtained in performance across prompts, though general trends reflect the same conclusions as Figure2 in the main text. That is, as negative labels are more similar to positive ones, verification accuracy decreases, and different model and prompt pairings trade off false positives and false negatives differently.

C.3 Few-Shot Exemplars

We experiment with zero-shot, one-shot, and three-shot prompts. From problems outside of our evaluation set, we pulled one problem from grades K-8 and three from HS. We pair these problems with one positive label and one negative one, which is a randomly selected, conceptually similar neighbor to the problem’s positive labels in the ATC map. We write Chain-of-Thought-like explanations for these problem and $\mathcal{S}$ pairs to create few-shot exemplars to insert into prompts (e.g. Example {i} in Figures 10-15 would be repeated for each exemplar). An example of a few-shot exemplar:

Problem:
You’re mixing ingredients for cookies. The recipe says to combine 6 tablespoons, or $\frac{3}{4}$ stick, of butter with 1 cup of sugar. You accidentally mix in a whole stick of butter (8 tablespoons) with the cup of sugar. How can you fix this?

Standard description:
Use ratio and rate reasoning to solve real-world and mathematical problems, e.g., by reasoning about tables of equivalent ratios, tape diagrams, double number line diagrams, or equations.

Answer:
yes

Explanation:
This problem solves a real-world problem involving mixing ingredients for cookies. A student doing this problem would need to reason about equivalent ratios of butter amounts to sugar amounts.

For one-shot prompts, we randomly select an exemplar problem and $\mathcal{S}$ pair from any grade. For three-shot prompts, we select exemplars that span wide and narrow grade ranges: {(K, 7, HS), (K, 3, 5), (6, 7, 8), (HS, HS, HS)}. We initially hypothesized that models may perform better on problems accompanied by few-shot exemplars in similar grade spans, but we did not confirm this hypothesis with our exemplar pool (Figure6).

C.4 Error Analysis

In the main paper, we discuss false negative and false positive patterns among models by referring to the language within particularly challenging $\mathcal{S}$. To do this analysis, we tokenize $\mathcal{S}$ using Bling Fire, and remove tokens that are less than 2 characters long, are nltk English stopwords, or appear in less than 5 $\mathcal{S}$. Table6 shows the results of this analysis.

C.5 Performance Across Grade Levels

Intuitively, higher grade levels may suggest lower verification performance, based on known measures of problem “hardness” Hase etal. (2024). However, Figure7 shows a lack of a consistently positive trend between F1 (calculated over all positive and negative examples) and problems’ grade level. For all 27 combinations of model and prompting approaches, we did not observe a significantly positive Spearman $\rho$ between performance and grade level, where significance is measured as $p<0.05$ with Bonferroni correction.

D.1 Prompts

We write 15 possible prompt templates $\{\mathcal{Q}_{i}\mid i=1,2,3,\ldots,15\}$ and run models on 500 random instances of a simple toy task to filter out catastrophic templates. In this toy task, we provide models an OER problem and 5 random $\mathcal{S}$ descriptions, and models are asked to select positive labels hidden among these options.

We observed models struggling to follow response formatting instructions in early experiments. Thus, our prompt paraphrases also vary specifications around response format, e.g. comma-separated selected options (Figure16). In addition, $\mathcal{P}_{7}$-$\mathcal{P}_{15}$ suggested walking through the steps of solving or completing a given problem (e.g. Figure22). To rank $\mathcal{Q}_{i}$ per model, we first calculate the rate to which models format responses correctly, and break ties based on models’ weak accuracy, where responses are correct if predictions overlap with gold labels.

Models, like with verification, vary in performance across all 15 $\mathcal{Q}_{i}$. On our toy task, we observe exact accuracy ranges of 0.046-0.332 for Llama 2, 0.304-0.542 for Mixtral, and 0.586-0.75 for GPT-4. This performance ordering of models parallels the one obtained for verification, though the gaps among models and prompts are greater here. Some $\mathcal{Q}_{i}$ favored by one model are highly detrimental to another, and asking a model to walk through doing a problem does not necessarily improve performance. Each models’ top-3 prompts are: $\mathcal{Q}_{3}$ (Figure17), $\mathcal{Q}_{11}$ (Figure22), $\mathcal{Q}_{12}$ (Figure23) for Llama 2, $\mathcal{Q}_{5}$ (Figure19), $\mathcal{Q}_{6}$ (Figure20), $\mathcal{Q}_{12}$ (Figure23) for Mixtral, and $\mathcal{Q}_{1}$ (Figure16), $\mathcal{Q}_{4}$ (Figure18), $\mathcal{Q}_{10}$ (Figure21) for GPT-4.

We ran these top-3 performing prompt templates for each model on the full evaluation set, and find that the trends we describe in the main text’s Figure3 generalize to other top-performing prompts (Figure8). That is, as stronger models are asked to make more granular decisions from $\mathcal{D}$ to $\mathcal{C}$ to $\mathcal{S}$, their performance decreases.

D.2 Few-Shot Exemplars

We again wrote explanations for $\mathcal{D}$-, $\mathcal{C}$-, and $\mathcal{S}$-level tagging decisions to create exemplars, and sampled one-shot and three-shot exemplars in the same manner as we did with verification. An example of an exemplar at the $\mathcal{S}$-level:

Problem:
Task
Cruz and Erica were both getting ready for soccer.
Cruz ran 1 lap around the school.
Erica ran 3 laps around the playground.
Erica said,
I ran more laps, so I ran farther.
Cruz said,
4 laps around the school is 1 mile, but it takes 12 laps around the playground to go 1 mile. My laps are much longer, so I ran farther.
Who is right? Draw a picture to help you explain your answer.

Options:
A. Explain why a fraction a/b is equivalent to a fraction (n × a)/(n × b) by using visual fraction models, with attention to how the number and size of the parts differ even though the two fractions themselves are the same size. Use this principle to recognize and generate equivalent fractions. Grade 4 expectations in this domain are limited to fractions with denominators 2, 3, 4, 5, 6, 8, 10, 12, and 100.
B. Compare two fractions with different numerators and different denominators, e.g., by creating common denominators or numerators, or by comparing to a benchmark fraction such as 1/2. Recognize that comparisons are valid only when the two fractions refer to the same whole. Record the results of comparisons with symbols >, =, or <, and justify the conclusions, e.g., by using a visual fraction model. Grade 4 expectations in this domain are limited to fractions with denominators 2, 3, 4, 5, 6, 8, 10, 12, and 100.

Answer:
A, B

Explanation:
This task asks students to compare two fractions that are equal. The two fractions have different numerators and denominators (e.g. 1/4 and 3/12). The students need to explain why the two fractions are equal by drawing a picture, which involves using visual fraction models.

E.1 Creating Prompt Templates

First, we asked a subset of teachers (our first cohort) to write prompts that reflect the following: Pretend you have 1-3 standards (not the MP standards) in mind that you want to teach. How would you ask a model to generate a math problem based on these standard/s? Here, “MP standards” refer to mathematical practice standards, while our work focuses on CCSS math content standards.

We found that teachers usually refer to standards by their labels, e.g. 4.NBT.A.1, when searching for curricular resources, rather than specify full standard descriptions. In addition, they do not always specify in their prompts that the standards are CCSS standards, rather than some other set of standards that may use similar labels (e.g. state-specific standards). Thus, at the end of each prompt template, we appended each CCSS standard’s natural language description to better guide models, e.g. Standard {label}: {description}.

Altogether, teachers wrote around 20 prompts, though due to variation in teachers’ prior experience with LMs, some prompts were not suitable as instruction-like inputs, e.g. I’d like to see if AI could quickly generate sets of problems with specific root types, such as integers, fractions, etc both with the coefficient of $x^{2}$ as 1 and with it as an integer. Thus, we only select only a subset of all proposed prompts, especially ones that could be minimially edited into prompt templates (Table8). We remove extraneous, standard-specific information, e.g. My students need to prove they can add and subtract within 10 using models, to make prompts generalizable for nearly all standards. Teachers’ suggested prompts varied in complexity, including simple requests such as Create a math problem that uses {$\mathcal{S}$} to ones that include details around their students’ needs, e.g. This problem will need to be simplified to support diverse learners in the resource room. The diversity within teachers’ suggested prompts, though, could be informative for future work that extends the evaluation of problem generation beyond template-based inputs.

We observed that some teachers wished to teach multiple standards within a single generated problem, but to scope this study, we simplify all prompt templates to include only a single standard. Thus, to make our setup reflective of standards that may be realistically taught in isolation within a problem, when sampling standards to insert into prompts, we chose ones that appear in MathFish with singly-labeled problems.

E.2 Other Observations of Generated Problems

Aside from standards alignment issues, teachers wrote down additional observations that pertain to the quality of generated problems. We outline two common ones here, in case they are informative for guiding more extensive future work.