Leveraging LLMs for Efficient and Accurate Data Management: model customisation

After applying LLM methods for annotation and classification out of the box in my previous article Leveraging LLMs for Efficient and Accurate Data Management, model customizing naturally becomes the next step to maximize detection accuracy. 

Data preparation

Each and every model need to fed with data, in correct format, amount and structure.

Series of 'Person names' as in my DataFrame is not enough to fine tune spaCy model. Names they should be explicitly labeled. Moreover they should be putted into context, into sentences and then labeled, so spaCy can understand the world in context. In my case, I used only the label 'Person', but other labels can be used as well.

df = pd.DataFrame(inventors, columns=["Inventor"])

In order to annotate the dataset at scale, I leveraged three approaches: spaCy, GPT-2, and Zephyr GenAI models.

spiCy with manual prompt

Creating a training set in spaCy starts with compiling a list of prompts, which should be predefined and then randomly enriched with names to generate a variety of labeled examples.

spicy_prompts = ["Assigned {name} to the ticket.",

"Spoke with {name} about the case.",

"{name} joined the weekly sync."]

spicy_training_data = []

for name in names:

    sentence = random.choice(spicy_prompts).replace("{name}", name) 

spicy_training_data.append((sentence, {"entities": [(sentence.index(name), start + len(name), "PERSON")]}))

GPT2 and Zephyr with generated prompts

Each and every model need to fed with data, in correct format, amount and structure.

Series of 'Person names' as in my DataFrame is not enough to fine tune spaCy model. Names they should be explicitly labeled. Moreover they should be putted into context, into sentences and then labeled, so spaCy can understand the world in context. In my case, I used only the label 'Person', but other labels can be used as well.

gpt_generator = pipeline("text-generation", model="gpt2")

gpt_training_data = []

for name in names:

    result = generate_sentence_with_annotation(name)    gpt_training_data.append(result)

zephyr_generator = pipeline("text-generation", model="HuggingFaceH4/zephyr-7b-beta")

zephyr_training_data = []

for name in names:

    result = generate_sentence_with_annotation(name)

zephyr_training_data.append(result)

def generate_sentence_with_annotation(name, max_length=50):

    prompt = name

    output = generator(prompt, max_length=max_length, num_return_sequences=1, do_sample=True, temperature=0.9)[0]['generated_text']

    sentence = output.split('.')[0].strip()

    start = sentence.find(name)

    if start == -1:

        return None

    end = start + len(name)

return (sentence, {"entities": [(start, end, "PERSON")]}

Wrapping prompts into spaCy format dataset: 

nlp = spacy.load("en_core_web_trf")

def save_to_spacy_format(gen_prompts, output_path):

nlp = spacy.blank("en")

doc_bin = DocBin()

for text, ann in gen_prompts:

    doc = nlp.make_doc(text)

    ents = []

    for start, end, label in ann["entities"]:

         span = doc.char_span(start, end, label=label)

         if span:

               ents.append(span)

               doc.ents = ent

           doc_bin.add(doc)

      doc_bin.to_disk(output_path)

Seems all ready to start training NER models on three spaCy-formatted datasets from spaCy, GPT-2, and Zephyr. 

Models training 

The config.cfg file in a spaCy project defines everything needed to train the NER model, including:

• [paths] Input .spacy files (train/dev)

• [nlp] Pipeline config (ner, tok2vec, etc.)

• [components.ner] NER component with model architecture

• [training] Epochs, optimizer, dropout, patience

• [corpora.train] Tells spaCy to read .spacy training files

• [system] Random seed and hardware config

To start train models:

datasets = {

    "spacy": "spacy_train_set.spacy",

    "gpt2": "gpt_train_set.spacy",

    "zephyr": "zephyr_train_set.spacy"} 

for name, dataset in datasets.items():

    output = f"./output_{name}"

    subprocess.run([

        "python", "-m", "spacy", "train", config.cfg,

        "--output", output,

        "--paths.train", dataset,

        "--paths.dev", dataset], check=True)

The output directory contains model-best and model-last subdirectories, where model-best holds the highest-scoring model ready for use, and model-last is useful for debugging or additional fine-tuning. 

Evaluation

The last important step is to determine which of the new models is most suitable for the goal, by comparing them to each other and to the original model.

The evaluation set can be prepared in a similar way to the training set. For the spaCy workflow, the prompts should be new,  while the GenAI models handle prompt variation on their side.

def evaluate_ner_model(nlp, sentences, target_label="PERSON"):

    correct = 0

    for name, sentence in sentences:

        doc = nlp(sentence)

        if any(ent.text == name and ent.label_ == target_label for ent in doc.ents):

            correct += 1

    return correct / len(sentences)

The heatmap shows that the Zephyr model delivered the best results, with GPT-2 ranking second. 

Concluding idea 

Fine-tuning LLMs with different dataset preparation strategies shows clear trade-offs between accuracy, speed, and control. Zephyr offers the highest accuracy, GPT-2 is nearly good and advantage of both is efficient training dataset creation.

By boosting LLMs through efficient data management and smart customization, we can achieve exceptional accuracy and propel the whole process to new heights.