Eric Spencer
projects
Eric Spencer worked at Loyola's ITS Service Desk for two years (2024-2025). Roughly 70% of the tickets there are the same five questions: connecting to LUC-Secure, a Duo push that does not arrive, no way into Sakai, printer credits that do not show up, Outlook re-asking for a sign-in. The answers all sit in the public TeamDynamix knowledge base, which people do not read, because reading a KB article is the last thing anyone wants to do while the laptop will not join the wifi. They want to ask a person.
This is a voice-first RAG chatbot that pulls from Loyola's ITS knowledge base and answers spoken questions out loud.
The stack is Python/FastAPI, FAISS for the vector index, sentence-transformers for embeddings, and Ollama running a local llama3.1:8b for inference. Speech-to-text is faster-whisper; the project started on Vosk and switched, for reasons below. Text-to-speech uses Microsoft Edge's free voices via edge-tts. The ingestion script crawls services.luc.edu/TDClient/33/Portal/KB/, strips boilerplate with trafilatura, chunks article text at about 900 characters with 120 of overlap, embeds each chunk, and writes a FAISS index to data/faiss/faiss.index. Retrieval at query time is a top-k similarity lookup whose results are stuffed into the system prompt. There is no reranker.
The whole thing runs in a Docker container. A heroku.yml covers the deploy path that was actually used. Heroku cannot run the local LLM, so that deployment falls back to a HuggingFace-hosted model (tiiuae/zephyr-7b-instruct) over the HF Inference API.
A few things were harder than expected.
Speech-to-text on technical jargon is rough. Vosk handled ordinary English fine, but "Duo" came back as "duel", "LUC-Secure" as "luxor", "Sakai" as "ski". faster-whisper replaced it: heavier, but it gets the proper nouns right most of the time. The trade is a couple of seconds of cold-start latency on a laptop CPU, which is noticeable in a spoken exchange. Fine-tuning on a small domain vocabulary was considered and never done.
The vector store candidates were Chroma, which wants a separate server process, Pinecone, which was more than the project needed and not free at the scale required, and FAISS, which is a file on disk. FAISS won as the simplest thing that could possibly work. The cost is no built-in metadata filtering, so filtering by KB category happens in Python after the search.
Local LLM latency on a MacBook is the bottleneck. llama3.1:8b under Ollama answers a typical query in maybe 4-8 seconds, which is long for someone sitting and waiting for the speaker to start. The HuggingFace remote path is faster but rate-limited, and it assumes an API key that not every user has. An engine dropdown selects between them.
The last rabbit hole was a full-duplex voice mode using NVIDIA's PersonaPlex 7B speech-to-speech model: no separate STT and TTS step, with the model hearing the speaker and replying directly. It loads on Apple Silicon via MPS; then came moshi-mlx, then the fact that the moshi-personaplex package pins torch<2.5, which locks the environment to Python 3.11. The commits from April are mostly attempts to make it work, and "wait personaplex won't work" is a real commit message. It is still in the repo as an optional engine, and the cascaded mode (Whisper, then the LLM, then Edge TTS) is the default.
ITS did not adopt it, and it was never pitched: by the time there was a version worth showing, graduation had already ended the service desk job. A current student desker can fork it and pitch it to Loyola. The ingestion script will pick up whatever is on the TDX portal at the time, and the knowledge base has not changed structure since.
Left to do: scoping retrieval to a single school, so that a question clearly about Sakai returns only Sakai articles; better handling of the escape hatch to a human; and possibly a hosted demo that avoids installing 15 GB of torch wheels.