flappyjet

Basicly, a LLM has dozens of transformer layers (attention layer + ffn layer). We cannot load the whole model data, or even a single transformer layer on NPU sram(~1MB). So, we put the whole model in RAM, transport a slice of data to NPU by command stream each time, then next slice. Several times of the model size may be transported between ddr and npu. This cost is huge so I still run LLM on pure cpu.

I agree with you. Neural network works well on NPU. I tried to load tiny llm model, but it's still too large for NPU. The bottleneck is the sram in NPU, not computing capacity. And I found an interesting project https://github.com/NotPunchnox/rkllama

What a nice picture! Would you be generous to post your NPU device info? I'm also looking for some w8a8 llm models working on NPU (the GEMM capability is perfect, isn't it).

Do you have rocket kernel with mesa that enable NPU working now? Or with rknn kernel? I'm a hobbyist working on a script to detect the NPU environment for ARM64 edge devices. Since I only have limited hardware, I’m looking for volunteers to run a quick test and see what is the environment on different NPU-equipped boards. This is a personal experimental project with no guarantees, just a survey to see what works and what doesn't. GitHub Repository: [npu-toolbox](https://github.com/flappyjet/npu-toolbox/) Test Script: `curl -sSL https://github.com/flappyjet/npu-toolbox/raw/refs/heads/main/scripts/npu_probe.sh | bash` Report Results: Please drop a comment in this survey issue: [Call for Testing: NPU detection Script](https://github.com/flappyjet/npu-toolbox/issues/1) Here's my A311d device info # npu-toolbox benchmark