Uttam Emmaniuel Dammu

This project presents quantitative benchmarking of Large Language Models (LLMs) to analyze how energy consumption, latency, and accuracy vary across open-source and closed-source systems. It highlights efficiency trade-offs among quantized open models such as Mamba-Codestral-7B and Qwen2-7B (4-bit and 8-bit precision) versus closed APIs like ChatGPT and Gemini. Open models were benchmarked locally using GPU telemetry and CodeCarbon, while closed APIs were tested through response-time monitoring. The study investigates how electricity usage changes with prompt size, how response time changes as token counts grow, and how efficient open models are per 1,000 tokens. Open models were measured locally with energy tracking, while closed API models are shown for latency context as energy is not observable. Energy consumption increases linearly with the number of input tokens. Mamba-7B (4-bit) demonstrates the lowest energy curve, while Qwen2-7B (8-bit) maintains a slightly higher curve, balancing precision and efficiency. Additional analysis includes energy per 1,000 tokens, latency distribution, and carbon emissions, demonstrating that reducing model precision and optimizing token throughput can significantly lower carbon emissions in real deployments.

This research is a continuation of my prior practicum project, which was an analysis of Large Language Model (LLM) inference energy efficiency, is the basis of this research. Previously benchmark demonstration on open and closed LLM’s on AWS cloud GPU infrastructure (L4 & A10G) resulted that prompt optimization can lower energy usage by up to 40% on cloud hardware without degrading any output quality. This current project aims to follow more towards resource-constrained edge computing, particularly on an NVIDIA Jetson Orin Nano device. Main research question is: Does the structural design of a user prompt varying in length, specificity, and reasoning strategy yield a statistically significant difference in energy consumption during LLM inference on an NVIDIA Jetson Nano. For this a controlled environment under TinyLlama-1.1B was used as the primary quantized model. Testing was done under four prompt styles: Structured, Concise, Verbose and Chain-of-Thought and evaluated across three benchmark categories: Q/A, Reasoning and Summarization. A modular automated pipeline was developed for dataset sampling, controlling the prompts, execution, cleaning, auditing and metric logging across runs. The main collection of metrics are Input/Output Tokens, Latency, Power and Total energy. An energy prediction model (a pre/post-regression model based on the use of prompts) has been established be improve our understanding of these patterns as to their influence on energy consumption. Additionally, exploratory analysis of prompt style, token count, latency and energy were performed. Prompt styling was found to indirectly influence energy through length and/or run time; however, latency was found to be the primary predictor of consumption. These findings back up previous assumptions regarding prompt design and its effect on energy consumption demonstrating that prompt design affects energy indirectly rather than from any other direct structural differences.