Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :')

Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions.

Perhaps SWE is dead after all, but LLMs didn't kill it...

> But we don't have a compelling enough theory yet to explain the RAM usage miracle.

My guess would be that the model is faulted into memory lazily page by page (4K or 16K chunks) as the model is used, so only the actual parts that are needed are loaded.

The kernel also removes old pages from the page cache to make room for new ones, and especially so if the computer is using a lot of its RAM. As with all performance things, this approach trades off inference speed for memory usage, but likely faster overall because you don't have to read the entire thing from disk at the start. Each input will take a different path through the model, and will require loading more of it.

The cool part is that this memory architecture should work just fine with hardware acceleration, too, as long as the computer has unified memory (anything with an integrated GPU). This approach likely won't be possible with dedicated GPUs/VRAM.

This approach _does_ still work to run a dense model with limited memory, but the time/memory savings would just be less. The GPU doesn't multiply every matrix in the file literally simultaneously, so the page cache doesn't need to contain the entire model at once.

Based on that discussion, it definitely sounds like some sort of bug is hiding. Perhaps run some evaluations to compare perplexity to the standard implementation?

Edit: looks like there's now confirmation that running it on a 10GB VM slows inference down massively, so looks like the only thing strange is the memory usage reading on some systems.

Took a look at it, did you try MAP_HUGETLB? This looks like the kind of application that can gain very large runtime advantages from avoiding TLB pressure. It might take a bit longer (or fail entirely) on machines where you can't get enough huge pages, but attempting it (or probing for free pages via /proc/meminfo) and then falling back to mapping it without might take slightly longer for the mmap() but the advantages of taking an order of magnitude (assuming you can get 1G pages) fewer TLB misses might be worth it.

Is the title misleading here ?

30B quantized requires 19.5 GB, not 6GB; Otherwise severe swapping to disk

  model    original size   quantized size (4-bit)
  7B     13 GB    3.9 GB
  13B    24 GB    7.8 GB
  30B    60 GB    19.5 GB
  65B    120 GB   38.5 GB

I’m hopeful that when especially skilled developers like you have banged on minimizing inference resources, the others and you will start looking at distributed training ideas. Probably there is a way to decentralize the training so we can all throw in our GPUs together on building the most useful models for code generation than can be free to use and relatively cheap to run inference on. If you have any thoughts on that side of the LLM space I’m sure we would all be super curious to hear them.

Thank you for the amazing work. It’s so appreciated by so many on HN like me I’m sure.

Why is it behaving sparsely? There are only dense operations, right?

Gosh, thank you for getting to this before I did. The first thing I said when I saw it loading tens of GB from the disk on each run is, is there some reason they're not using mmap?

This isn't just a matter of making the 30B model run in 6GB or whatever. You can now run the largest model, without heavy quantization, and let the OS figure it out. It won't be as fast as having "enough" memory, but it will run.

In theory you could always have done this with swap, but swap is even slower because evictions have to be written back to swap (and wear out your SSD if your swap isn't on glacially slow spinning rust) instead of just discarded because the OS knows where to read it back from the filesystem.

This should also make it much more efficient to run multiple instances at once because they can share the block cache.

(I wonder if anybody has done this with Stable Diffusion etc.)

It really shouldn't act as a sparse model. I would bet on something being off.

Thanks for this! I was able to integrate alpaca-30B into a slack bot & a quick tkinter GUI (coded by GPT-4 tbh) by just shelling out to `./main` in both cases, since model loading is so quick now. (I didn't even have to ask GPT-4 to code me up Python bindings to llama's c-style api!)

Great work. Is the new file format described anywhere? Skimming the issue comments I have a vague sense that r/o matter was colocated somewhere for zero copy mmap or is there more to it?

Just shows how inefficient some of the ML research code can be

Have you tried running it against a quantized model on HuggingFace with identical inputs and deterministic sampling to check if the outputs you're getting are identical? I think that should confirm/eliminate any concern of the model being evaluated incorrectly.

Maybe off topic, but I just wanted to say that you're an inspiration!

This is nothing short of legendary. Was following the thread on Twitter and LOLed at the replies of “Praise be Jart”, but there’s something of the sublime here. Great weight wrangling judo :)

Thanks for all the nice projects you are releasing for free, btw.

Authorship of this change is contested.

https://news.ycombinator.com/item?id=35431865

Some OS’s zram compress the unpinned pages instead of swapping to disk. It might be faster than fetching the pages again from disk. I wonder if this is a reason why folks see different results.

>I'm glad you're happy with the fact that LLaMA 30B (a 20gb file) can be evaluated with only 4gb of memory usage!

Isn't LLaMA 30B a set of 4 files (60,59Gb)?

-edit- nvm, It's quantized. My bad

Before, it had to read the 16GB into a buffer, which itself could be written to disk if the system needs to page out.

That is a big reason startup time is fast.

Could mmap also be used to improve the memory usage of whisper.cpp?

How diverse is the training corpus?

Hey, I saw your thoughtful comment before you deleted it. I just wanted to apologize — I had no idea this was a de facto Show HN, and certainly didn’t mean to make it about something other than this project.

The only reason I posted it is because Facebook had been DMCAing a few repos, and I wanted to reassure everyone that they can hack freely without worry. That’s all.

I’m really sorry if I overshadowed your moment on HN, and I feel terrible about that. I’ll try to read the room a little better before posting from now on.

Please have a wonderful weekend, and thanks so much for your hard work on LLaMA!

EDIT: The mods have mercifully downweighted my comment, which is a relief. Thank you for speaking up about that, and sorry again.

If you'd like to discuss any of the topics you originally posted about, you had some great points.

Maybe not so fast. Other users are reporting that it’s not actually running properly in environments with limited RAM. The reduced memory usage might be more of a reporting misunderstanding, not an actual reduction in memory usage.

>What has everyone been doing wrong all these years

So it's important to note that all of these improvements are the kinds of things that are cheap to run on a pretrained model. And all of the developments involving large language models recently have been the product of hundreds of thousands of dollars in rented compute time. Once you start putting six digits on a pile of model weights, that becomes a capital cost that the business either needs to recuperate or turn into a competitive advantage. So everyone who scales up to this point doesn't release model weights.

The model in question - LLaMA - isn't even a public model. It leaked and people copied[0] it. But because such a large model leaked, now people can actually work on iterative improvements again.

Unfortunately we don't really have a way for the FOSS community to pool together that much money to buy compute from cloud providers. Contributions-in-kind through distributed computing (e.g. a "GPT@home" project) would require significant changes to training methodology[1]. Further compounding this, the state-of-the-art is actually kind of a trade secret now. Exact training code isn't always available, and OpenAI has even gone so far as to refuse to say anything about GPT-4's architecture or training set to prevent open replication.

[0] I'm avoiding the use of the verb "stole" here, not just because I support filesharing, but because copyright law likely does not protect AI model weights alone.

[1] AI training has very high minimum requirements to get in the door. If your GPU has 12GB of VRAM and your model and gradients require 13GB, you can't train the model. CPUs don't have this limitation but they are ridiculously inefficient for any training task. There are techniques like ZeRO to give pagefile-like state partitioning to GPU training, but that requires additional engineering.

Roughly: OpenAIs don’t employ enough jarts.

In other words, the groups of folks working on training models don’t necessarily have access to the sort of optimization engineers that are working in other areas.

When all of this leaked into the open, it caused a lot of people knowledgeable in different areas to put their own expertise to the task. Some of those efforts (mmap) pay off spectacularly. Expect industry to copy the best of these improvements.

> but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks?

It’s several things:

* Cutting-edge code, not overly concerned with optimization

* Code written by scientists, who aren’t known for being the world’s greatest programmers

* The obsession the research world has with using Python

Not surprising that there’s a lot of low-hanging fruit that can be optimized.

I have predicted that LLaMA will be available on mobile phones before the end of this year. We are very close.

And THIS is why I always advocate for democratization of AI models, getting these into hands of open source community and letting people use it as they wish etc.

But a lot of people would rather only have govt or corp control of it...

The apparent answer is here: https://news.ycombinator.com/item?id=35398012

> mmap-ed memory pages backed by a file that aren't dirty aren't counted in an process's RSS usage, only kernel page cache. The mmap-ed regions of virtual memory does get counted in VSZ (virtual memory) but that is just virtual and can be larger than RAM+swap.

> I might be missing something but I actually couldn't reproduce.

Someone in the GitHub comments had the same experience when using a 10GB VM to limit memory usage.

It appears the claims of memory reduction were premature. Perhaps an artifact of how memory usage is being reported by some tools.

Same, performance of the quantised 30B model on my m1 16GB air is absolutely terrible. A couple of things I noticed on activity monitor: 1. "memory used" + "cached files" == 16GB (while swap is zero) 2. Disk reading is 500-600MB/s 3. it seems that every token is computed exactly _after every ~20GB read from disk_ which actually points that for calculating each token it actually re-reads the weights file again (instead of caching it). I actually suspect that swapping may have been more efficient.

The last part (3) that it rereads the whole file again is an assumption and it could just be a coincidence that the new token is computed at every ~20GB read from disk, but it makes sense, as I do not think swapping would have been that inefficient.

Same here M1 MacBook Pro. Zero speed up on loading and inference.

Are the weights on NVME? Old SSD? HDD?

Now it's a C mess!

How so?

I've done some experiments here with Llama 13B, in my subjective experience the original fp16 model is significantly better (particularly on coding tasks). There are a bunch of synthetic benchmarks such a wikitext2 PPL and all the whiz bang quantization schemes seem to score well but subjectively something is missing.

I've been able to compare 4 bit GPTQ, naive int8, LLM.int8, fp16, and fp32. LLM.int8 does impressively well but inference is 4-5x slower than native fp16.

Oddly I recently ran a fork of the model on the ONNX runtime, I'm convinced that the model performed better than pytorch/transformers, perhaps subtle differences in floating point behavior etc between kernels on different hardware significantly influence performance.

The most promising next step in the quantization space IMO has to be fp8, there's a lot of hardware vendors adding support, and there's a lot of reasons to believe fp8 will outperform most current quantization schemes [1][2]. Particularly when combined with quantization aware training / fine tuning (I think OpenAI did something similar for GPT3.5 "turbo").

If anybody is interested I'm currently working on an open source fp8 emulation library for pytorch, hoping to build something equivalent to bitsandbytes. If you are interested in collaborating my email is in my profile.

1. https://arxiv.org/abs/2208.09225 2. https://arxiv.org/abs/2209.05433

For this specific implementation here's info from llama.cpp repo:

Perplexity - model options

5.5985 - 13B, q4_0

5.9565 - 7B, f16

6.3001 - 7B, q4_1

6.5949 - 7B, q4_0

6.5995 - 7B, q4_0, --memory_f16

According to this repo[1] difference is about 3% in their implementation with right group size. If you'd like to know more, I think you should read GPTQ paper[2].

[1] https://github.com/qwopqwop200/GPTQ-for-LLaMa

[2] https://arxiv.org/abs/2210.17323

Define "comprehensive?"

There are some benchmarks here: https://www.reddit.com/r/LocalLLaMA/comments/1248183/i_am_cu... and here: https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-i...

Check out the original paper on quantization, which has some benchmarks: https://arxiv.org/pdf/2210.17323.pdf and this paper, which also has benchmarks and explains how they determined that 4-bit quantization is optimal compared to 3-bit: https://arxiv.org/pdf/2212.09720.pdf

I also think the discussion of that second paper here is interesting, though it doesn't have its own benchmarks: https://github.com/oobabooga/text-generation-webui/issues/17...

Some results here: https://github.com/ggerganov/llama.cpp/discussions/406

tl;dr quantizing the 13B model gives up about 30% of the improvement you get from moving from 7B to 13B - so quantized 13B is still much better than unquantized 7B. Similar results for the larger models.

It's not a bug, but it's misreading the htop output as mmap doesn't show up as a resident set size there. The pages are RO and not dirty so it's "on the OS" to count it and the OP had lots of RAM on the computer so the model just resides in his page cache instead.

yeah, I believe some readers are misinterpreting the report. The OS manages mmap, it won't show up as "regular" memory utilization because it's lazy-loaded and automatically managed. If the OS can keep the whole file in memory, it will, and it will also magically swap to disk prioritizing explicit memory allocation (malloc).

Sounds like the big win is load time from the optimizations. Also, maybe llama.cpp now supports low-memory systems through mmap swapping? ... at the end of the day, 30B quantized is still 19GB...

Maybe lots of the data is embedding values or tokenizer stuff, where a single prompt uses a fraction of those values. And then the rest of the model is quite small.

If you read a file with malloc and memcpy, it copies the data from the kernel to userspace. With mmap there is no copying.

Before ChatGPT was in beta, there were already models that fit into 2gb and smaller. They were complete shit, but they did exist.

Yes, if the model is mmap'ed read-only (as I'm sure it is).

There are other bottlenecks than CPU cores though, it might not be very useful to run multiple in parallel..

With all my dislike to Electron, I struggle to remember even one Electron app that managed to use 6 gigs.

Maybe they should do more mmap as well.

LMAO

I doubt it: text size and text pattern size don't scale linearly.

According to https://mobile.twitter.com/JustineTunney/status/164190201019... you can probably use the conversion tools from the repo on Alpaca and get the same result.

If you want to run larger Alpaca models on a low VRAM GPU, try FlexGen. I think https://github.com/oobabooga/text-generation-webui/ is one of the easier ways to get that going.

No(llama.cpp is cpu-only) and no(you need to requantize the model).

"Could LLM eventually replace Google"

If you try to use LLMs as a Google replacement you're going to run into problems pretty quick.

LLMs are better thought of as "calculators for words" - retrieval of facts is a by-product of how they are trained, but it's not their core competence at all.

LLaMA at 4bit on my laptop is around 3.9GB. There's no way you could compress all of human knowledge into less than 4GB of space. Even ChatGPT / GPT-4, though much bigger, couldn't possible contain all of the information that you might want them to contain.

https://www.newyorker.com/tech/annals-of-technology/chatgpt-... "ChatGPT Is a Blurry JPEG of the Web" is a neat way of thinking about that.

But... it turns out you don't actually need a single LLM that contains all knowledge. What's much more interesting is a smaller LLM that has the ability to run tools - such as executing searches against larger indexes of data. That's what Bing and Google Bard do already, and it's a pattern we can implement ourselves pretty easily: https://til.simonwillison.net/llms/python-react-pattern

The thing that excites me is the idea of having a 4GB (or 8GB or 16GB even) model on my own computer that has enough capabilities that it can operate as a personal agent, running searches, executing calculations and generally doing really useful stuff despite not containing a great deal of detailed knowledge about the world at all.

The largest LLaMA model at ~60 billion parameters is not quite as large as ChatGPT 3 in size, and probably not quite as well trained, but it's basically in the same class. Even the complete, not quantitized model, can be run with llama.cpp on ARM64 and x86_64 CPUs already, assuming you have enough RAM (128 GB?).

Minor correction, chatGPT uses GPT-3.5 and (most recently, if you pay $20/month) GPT-4. Their branding definitely needs some work haha. We are in track for you to be able to run something like chatGPT locally!

worse, yes, yes, no

Windows Server could use up to 64 GB for a 32-bit operating system. Individual processes couldn't map more than 4 GB, but the total could be larger: https://en.wikipedia.org/wiki/Physical_Address_Extension

It actually happened. With paging and bigger word sizes, which were common in 32 bit systems even with hardware acceleration.

A 32bit address space only means you have 4GibiAddresses, which do not need to be pointing to single bytes. In fact the natural thing to do in a 32 bit system for a structure like this is moving 32bit words, which actually means you're addressing a 16GB space, flat. And then there's segmentation.

For instance the 286 had a 24bit address showing for 16MB in direct addressing mode and 1GB via segmentation (what back then was usually referred to by virtual memory)

The 386 had a 32 bit address width and its MMU allowed access to 64TB in virtual mode and 4GB in protected mode.This was indeed one of the reasons Linux was not made 286-compatible. Its protected mode was only 1GB and segmented rather than 4GB flat, so Linus didn't have to deal with XMS or EMS for a chip that was becoming obsolete soon anyway. But the 1GB space was there, and at the time that was plenty.

Not yet. GPT-4 helped answer some questions I had about the WIN32 API but that's the most use I've gotten out of it so far. I'd love for it to be able to help me more, and GPT-4 is absolutely 10x better than GPT 3.5. But it's just not strong enough at the kinds of coding I do that it can give me something that I won't want to change completely. They should just train a ChatJustine on my code.

Using a memory mapped file doesn't use swap. The memory is backed by the file that is memory mapped!

Is llama open source? I heard it was pirated from Facebook

llama can run on an m1. T5 still needs a specialized gpu

CPU inference runs perfectly fine, though.

I was thinking something similar. Turns out that you don't need all the weights for any given prompt.

> LLaMA 30B appears to be a sparse model. While there's 20GB of weights, depending on your prompt I suppose only a small portion of that needs to be used at evaluation time [...]

Found the answer from the author of this amazing pull request: https://github.com/ggerganov/llama.cpp/discussions/638#discu...

Parameters have been quantized down to 4 bits per parameter, and not all parameters are needed at the same time.

https://github.com/oobabooga/text-generation-webui

I'd assume that 33B model should fit with this(only repo that I know of that implements SparseGPT and GPTQ for LLaMa), I, personally, haven't tried though. But you can try your luck https://github.com/lachlansneff/sparsellama

Yes, the 30B model is working for me on Windows 10 / AMD 5600G CPU / 32GB RAM, with llama.cpp release master-3525899 (already one release out of date!), in PowerShell, using the Python 3.10 version that automatically installs when you type "python3".

I did the following:

1. Create a new working directory.

2. git clone https://github.com/ggerganov/llama.cpp

3. Download the latest release from https://github.com/ggerganov/llama.cpp/releases (note the CPU requirements in the filename) and unzip directly into the working directory's llama.cpp/ - you'll have the .exe files and .py scripts in the same directory.

4. Open PowerShell, cd to the working directory/llama.cpp, and create a new Python virtual environment: python3 -m venv env and activate the environment: .\env\Scripts\Activate.ps1

5. Obtain the LLaMA model(s) via the magnet torrent link and place them in the models directory. I used 30B and it is slow, but usable, on my system. Not even ChatGPT 3 level especially for programming questions, but impressive.

6. python3 -m pip install torch numpy sentencepiece

7. python3 convert-pth-to-ggml.py models/30B/ 1 (you may delete the original .pth model files after this step to save disk space)

8. .\quantize.exe ./models/30B/ggml-model-f16.bin ./models/30B/ggml-model-q4_0.bin 2

9. I copied the examples/chat-13B.bat to a new chat-30B.bat file, updated the model directory, and changed the last line of the script to: .\main.exe

10. Run using: .\examples\chat-30B.bat

https://github.com/ggerganov/llama.cpp#usage has details, although it assumes 7B and skips a few of the above steps.

I haven't been able to run it myself yet, but according to what I read so far from people who did, the 30B model is where the "magic" starts to happen.

Check out the graph on page 3 of this PDF: https://arxiv.org/abs/2302.13971 The 33B model started beating the 7B when it had been trained on only 1/3 as much data. And then they kept training it to 40% more than the total that 7B was trained on. It's better.

lesmo provides a magnet uri to all models here. it's inside (499kb): https://github.com/facebookresearch/llama/pull/73/files/016a...

It is as good as GPT-3 at most sizes. Instruct layer needs to be put on top in order for it to compete with GPT 3.5(which powers ChatGPT). It can be done with comparatively little amount of compute(couple hundred bucks worth of compute for small models, I'd assume low thousands for 65B).

I assume the former is just the foundation model (which only predicts text) while the latter is instruction tuned.

Where are you going to download that from?

IANYL - This is not legal advice.

As you may be aware, a counter-notice that meets the statutory requirements will result in reinstatement unless Meta sues over it. So the question isn't so much whether your counter-notice covers all the potential defenses as whether Meta is willing to sue.

The primary hurdle you're going to face is your argument that weights are not creative works, and not copyrightable. That argument is unlikely to succeed for the the following reasons (just off the top of my head): (i) The act of selecting training data is more akin to an encyclopedia than the white pages example you used on Twitter, and encyclopedias are copyrightable as to the arrangement and specific descriptions of facts, even though the underlying facts are not; and (ii) LLaMA, GPT-N, Bard, etc, all have different weights, different numbers of parameters, different amounts of training data, and different tuning, which puts paid to the idea that there is only one way to express the underlying ideas, or that all of it is necessarily controlled by the specific math involved.

In addition, Meta has the financial wherewithal to crush you even were you legally on sound footing.

The upshot of all of this is that you may win for now if Meta doesn't want to file a rush lawsuit, but in the long run, you likely lose.

All models trained on public data need to be made public. As it is their outputs are not copyrightable, it’s not a stretch to say models are public domain.

Thank you for putting your ass on the line and deciding to challenge $megacorp on their claims of owning the copyright on NN weights that have been trained on public (and probably, to some degree, also copyrighted) data. This seems to very much be uncharted territory in the legal space, so there are a lot of unknowns.

I don't consider it ethical to compress the corpus of human knowledge into some NN weights and then closing those weights behind proprietary doors, and I hope that legislators will see this similarly.

My only worry is that they'll get you on some technicality, like that (some version of) your program used their servers afaik.

Wish you all luck in the world. We need much more clarity in legal status of these models.

Even if using LLaMA turns out to be legal, I very much doubt it is ethical. The model got leaked while it was only intended for research purposes. Meta engineered and paid for the training of this model. It's theirs.

Read readme in repo.