The models above are now available in the app

Added phi-2 and f16 tinyllama results. Also checked the prev results are not changed in commit 0e18b2e.

Text Generation speed using Mistral is more than useable on newer iPhones it seems. Prompt processing is very slow however, even when using Metal. I wonder if this is a compute or bandwidth limitation.

What's the minimum t/s for it to be usable? For example, is 9 t/s usable for Mistral 7b?

It's a UX design principle. For a model to be usable, the benchmark I've been working towards is 400ms for the first token and subsequent tokens. If a token isn't being processed to the user at least every 400ms, then the model won't be usable. Minimum usability should, therefore, be about 3t/s...

Thanks - the iPhone 15 Pro Max should be using the same A17 chip as iPhone 15 Pro, correct? At least this is what I get from wikipedia and the numbers seem to mostly match the one from @jhen0409 above

Yes. Both iPhone 15 Pro and Pro Max use A17. Concretely, it is named with A17 Pro by Apple. See here.

@ggerganov Just updated F16 results. The model is taken from https://huggingface.co/SergiusFlavius/TinyLlama-1.1B-1T-OpenOrca-GGUF/blob/main/tinyllama-1.1b-1t-openorca.F16.gguf

Add phi-2 3B Q4_0 results. Others can't be loaded.

Apple TV 4K (3rd gen) seems to be only one that has A15 (5CPU + 5GPU).
I checked my Apple TV 4K, and unfortunately, it is 2nd gen (A12) 😂

Damn, first LLM on a TV 😄

if you really need this, i can build and run these tests for aTV 3gen )

Don't really need it, but it might be a cool achievement. LLM on a watch has already been demonstrated: https://twitter.com/shxf0072/status/1736713832045982040, but I don't think that is the case for LLM on a TV

Thanks! I guess Mistral 7B does not fit on this device?

@ggerganov It does, I can load it and run it by sending a message, but the Bench button keeps aborting indicating heat up time being too long.

It's tokens per second.

thanks for confirmation

https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v0.3-GGUF/tree/main

The above model got loaded on my device (iphone 13, ios 17.3). But when I try to send a message or benchmark it, I get Heat up time is too long message. The same message I am getting in iphone 15 (simulator), iphone 15 pro (simulator), and iphone 15 pro max (simulator) as well. The size of the model is 2.05 GB. @ggerganov

Did you checkout code from 0e18b2e?
Did you enable release build? IIRC, I got some errors if not on release build.

Thanks I'm on the latest master branch as of February 2, and the issue still occurs. I opened the project directly with Xcode, configured the signing settings, and the issue appeared—all settings are at their defaults. Are there any additional steps or guidance?

I was able to run it, thank you. The instructions for running it are here: #4508

why master branch's example/llama.swiftui not supporting simulator, like Building for 'iOS-simulator', but linking in dylib (/opt/homebrew/Cellar/llama.cpp/4733/lib/libggml.dylib) built for 'macOS'

i want to use qwen2 which is not supported in 0e18b2e branch, is there any solution

@HaohongLin1 : Please guide me through the steps to build the LLAMA.cpp project. I can’t run LLAMA.swiftUI because I keep getting the error “llama.h not found.”

@tuan88it

OS: MacOS Sequoia 15.1.1

For the LLAMA.swiftUI, I only follow Instructions in this page, I don't need any extra steps to build it in XCode.

@HaohongLin1 : Please guide me through the steps to build the LLAMA.cpp project. I can’t run LLAMA.swiftUI because I keep getting the error “llama.h not found.”

git checkout 0e18b2e I can run the code , but I still face the error "error loading model: create_tensor: tensor 'output.weight' not found" 😭

After checkout I run the code just fine for both simulator and on device. Did you tap download link below first? (It takes a while to download some model).

Thanks for your work on this! I'm a definite noob but love the fact that I can study the code and read all the comments on this repo. Exciting that the attitude is 'let's see what we can do, and how to make it smaller/faster/better!

I’d like to ask how you built the iOS target. Does the memory information show up when running the benchmark? When I tested on Android, I found that the memory usage (PSS value) was more than twice the size of the model.

I just followed the instruction on this thread. For memory, it's a spec of the phone I'm running with.
The result os copied from running the benchmarks.