Why is fine tuning a chat model harder than fine tuning an autocomplete model?

[Harrolee]

Hey! n00b question here.

I noticed that Refact does not support fine tuning for chat models:

Why is it more difficult to fine tune a chat model than a fim/autocomplete model? Is it because the autocomplete functionality is achieved by changing the data format? And fine tuning a chat model would require adding a chat LoRA after the "new-content" fine tuning?

My probably-wrong understanding of how this works

Chat fine-tuning process, using LoRA:

1. get pre-trained model (e.g. llama2, deepseekcoder)
2. self-supervise fine tune* the pre-trained model on new data using LoRA
3. get a chat LoRA for the pre-trained model
4. apply chat LoRA and the LoRA from step 2 onto the pre-trained model

* my probably-wrong understanding of self-supervised fine tuning is:

1. format new data as a single text column with n rows of roughly similar length, like: {"text": "string content of data whose length is about the same for each entry"}
2. forward pass, back prop
3. evaluate loss by testing the model's accuracy in reproducing the value of each "text" entry

[olegklimov]

The trick is where to get new data for chat from. If all you have is company's codebase, you need to translate it into conversations similar to that at test time.

We are working on "self-play" that our system generates tasks for itself, but it's not as straightforward as finetune on data you see locally.

[Harrolee]

Ah I understand.
I had hoped that a LoRA would "teach" the LLM how to answer Chat queries about the unstructured text that it had been fine tuned on.

idea for generating data in Chat format from a codebase:

1. Chunk the codebase into functions or classes intelligently
2. Use something like Paul Gauthier's grep-ast to generate context around the chunk
3. Use that context to generate a prompt for an LLM. For each chunk, ask the LLM to generate n questions that could be asked about the codechunk, and use one-shot prompting to get the LLM to format its answer in a json format
4. Ask a second LLM to check that the format matches what we want
5. Append the result to a file

There's also a paper (that I can't seem to find atm) about creating conversational chat entries from git commits. Most git commits are not very good. I suppose an LLM could generate git commits from git history, but each additional step in this chain of Chat-content generation increases likelihood of failure.