Eric Spencer
projects
COMP 388 at Loyola is the special-topics CS course, whose subject changes with the professor. The Spring 2026 version was on large language models, which overlaps with the TLA+ and fine-tuning work at ai4fm. This repository holds two of its assignments.
The first assignment loads a Hugging Face model and runs it. The script llm_prompt.py loads gpt2 via transformers, takes a prompt off the command line, and does manual token-by-token generation rather than calling .generate(). The reason is environmental: on an M1 Mac, .generate() crashed with a bus error about half the time, and so did single-word prompts. The workaround is to force CPU-only execution, set PYTORCH_ENABLE_MPS_FALLBACK=1, and hand-roll the sampling loop.
The assignment then asked for a simple evaluation, done two ways. The first is a hand-written True/False dataset of 10 Wikipedia-style facts (evaluate_wiki_tf.py), on which GPT-2 scored 6 out of 10; every item it got wrong was a false statement that it called true. The second is BoolQ from Hugging Face datasets, 20 examples, run twice, once with the supporting passage and once without (evaluate_boolq.py). Both runs scored 45.0%. The outputs show GPT-2 answering yes to nearly everything, so the passage made no difference.
The second assignment was to pick a small open model that ships both a base and an instruction-tuned variant, compare them, and then try few-shot prompting on the base model. The models used are Qwen2.5-0.5B and Qwen2.5-0.5B-Instruct: same architecture, same tokenizer, differing only in the instruction-tuning step.
The base-vs-chat comparison behaved as expected. The base model treats a prompt as a document to continue, so "What is the capital of France?" produces more questions or bullet points, while the chat model answers "The capital of France is Paris."
The evaluation was on SNLI (natural language inference), 100 examples, greedy decoding:
The base model beat the chat model zero-shot, and few-shot prompting made the base model worse rather than better. One reading is that at 0.5B the instruction tuning overfits to chatty response shapes that hurt on a constrained-label task like NLI, and that the longer few-shot prompts push the small model past what it handles well. The few-shot run also hit CPU and memory ceilings, since the in-context examples triple the prompt length; a GPU or a quantized model would have finished cleanly.
This is coursework rather than a product, and the repository states its constraints plainly: pinned to Python 3.11, macOS on M1, CPU-only, with hardcoded paths that the assignment template called for.