The federal government faces a significant challenge in unlocking the full potential of artificial intelligence (AI) due to a lack of high-quality, diverse datasets necessary for training AI models. Despite having access to large amounts of data, finding and preparing relevant data is a critical hurdle. This challenge is not just a technical obstacle—it’s a strategic one, limiting the government’s ability to use AI effectively and maintain its global competitive edge. As demand for advanced AI applications increases, so does the need for efficiently developed training datasets. Recognizing this, there’s a growing focus on innovative approaches, such as using Large Language Models (LLMs) and variations of Large Multimodal Models (LMMs), to improve how we generate and assess training data at scale. As we look toward the future of training data development, these technologies can revolutionize the way we curate data, and accelerate the pace at which AI can be adopted. This single advancement could help the U.S. government not only keep pace with, but lead the global AI race. This article introduces what language models are, how they work, and how to choose from them based on need; and ultimately, shine light on how LLMs can transform the future of data labeling.
Brief Overview: GAI, LLMs & LMMs and how they work
Generative AI (GAI) harnesses deep learning to create content spanning text, images, code, and more, emulating human creativity by analyzing vast quantities of open-source data. Within this sphere, Large Language Models (LLMs) like OpenAI’s GPT series, LLaMA2, Mistral 7B, and the open-source OLMo4 stand out, employing transformer architectures for superior natural language processing capabilities, including text translation, summarization, and analysis1. These models undergo unsupervised learning from large text corpora, gaining a deep understanding of language semantics, syntax, and context. They identify patterns and relationships, allowing for sophisticated language tasks to be performed with near-human accuracy. Large Multimodal Models extend these abilities to interpret and integrate data from various modalities such as images, audio, and video, facilitating multifaceted information processing. LLMs utilize transformer neural networks to enhance sequence data processing and prioritize critical input features to generate better and more relevant outputs that infuse automation and human-machine workflows. Analyzing word sequence probabilities empowers LLMs to offer a wide array of language services, including creative writing and coding, reflecting the intricate blend of fluency and innovation that characterizes these models despite their significant training costs.
The effectiveness of Large Language Models (LLMs) hinges not just on the data they are trained on, but critically on how they are prompted to understand and respond to specific contexts. This process, known as Prompt Engineering, involves crafting queries or instructions (prompts) that guide the model in producing the desired outcomes. See Fig 1. The skill in formulating these prompts dramatically dictates the accuracy and relevance of the results. Techniques such as Zero Shot, One Shot, and Few Shot prompting allow for varying levels of specificity in guiding the model towards the precise information needed5. The power of LLMs is rooted in their comprehensive training across a vast array of topics, empowering them to generate contextually relevant content based on the nuanced prompts they receive. This capability enables users to tailor their queries interactively, effectively making LLMs highly versatile tools for extracting detailed information and generating context in the form of natural human language.
Choosing the right type of LLM for the task
Choosing the right Large Language Model (LLM) for your project requires understanding your goals, such as the tasks to be done, data involved, and needed outputs. You also need to weigh the costs and customization efforts against your organization’s budget and time constraints. Thinking through the entire project is crucial as you need to select the LLM that best suits your project needs, can evolve, and maximizes your return on investment. Common types of language models to explore include2:
- Language Representation Model (LRM), Zero Shot models, Single and Few Shot models5:
- LRMs are designed to generate language built on Pre-Trained Transformers with a large extensive corpus of text and contextual relationships.
- Zero Shot Models can make generalizations and perform tasks with minimal to no training or fine-tuning.
- Single & Few Shot Models require one or more examples to provide instructions to the model to generate relevant outputs.
- Multimodal models:
- LMMs are created to understand and leverage multiple kinds of data sources including text, imagery, audio, video, and even irregular sensor data.
- The key advantage of LMMs is their ability to make sense of multiple data types and understand the human language used to tie these modalities together simultaneously.
- Tunable and domain-specific models:
- Open source and fine-tunable models enable more precise outputs for a particular subject matter.
- An example could be a GPT-3 trained model geared toward medical understanding. The benefit would be the amount of data required to train is less than a general-purpose model and allows for more accurate results per question.
In addition, when deciding between large and small language models, it’s important to recognize that larger models come with significantly more parameters. This translates to a greater need for computational resources such as memory, storage, and processing power, both for training and operation. Consequently, larger models are not only more costly but also more complex to train and deploy, requiring substantial hardware, software, and expertise. Therefore, you must carefully assess the task’s complexity, the level of specificity required for your organization’s goals, and the available budget and the computational limitations of your organization. These factors can help guide your decision on the appropriate type of language model for an effective implementation.
Revolutionizing Data Labeling with LLMs
It is widely understood that optimal training datasets are based on ground truth, curated under human supervision, and have been meticulously labeled by subject matter experts to reflect precise and known characteristics. The down side is the process is costly and demands considerable time to ensure the data’s consistency and accuracy. The primary issue is the intensive demand for resources that hamper the speed and scalability of deploying sophisticated AI models. To combat the scale problem, incorporating Large Language Models into the process, could be the key ingredient to accelerate the creation of high-quality training data, minimizing the need for extensive domain expertise. A report by refuel corporation claims that automatic data labeling with a GPT4 model can label better than a human based on speed, accuracy and cost3 giving credence to this approach. Compared to the manual process of data labeling, there is a huge output advantage by using LLMs than without. Snorkel, another leading tool in this area, claims these methods can enhance productivity by as much as 10X6.
Fully automating the creation of synthetic and labeled data via LLMs is compelling, however the approach relies heavily on the quality of prompts used. Other research indicates that this method often falls short of expectations, hampered by the LLMs’ inherent biases, inconsistency in outputs, “hallucinations”, and the complexity of validating or adjusting model parameters at scale7. Additionally, memory limitations of LLMs require additional steps, storage and code to store outputs locally and fine tune the model as labels improve making the process more complex and prone to error. Incorporating feedback from domain experts on labels generated by Large Language Models (LLMs) alongside sophisticated prompt engineering techniques offers a promising compromise between human and fully automated solutions to maximize the production of high-quality datasets. This method effectively mitigates biases and errors inherent in automated processes using human-in-the-loop (HITL) to verify quality is maintained throughout the process. If LLM-generated labels are carefully reviewed and tuned under manual supervision against a ‘ground truth’ benchmark, then that effectively serves as a check against expected automation errors illustrated in Figure 2. Further, refining this process via iteration, provides a reinforced learning element that continuously improves the labels quality, and reduces errors in an annotation production workflow.
Iteratively tuning the LLM instruction prompts against ground truth, reduces the demand for manual intervention over time and improves the consistency, quality and scale of the labels produced. To accomplish this refinement process, identifying the right tools for the job will be the first step. Open-source tools such as Label Studio and Vellum provide commercial options for assisted labeling allowing you to create more accurate, unbiased, and high-quality datasets that are essential for training robust AI models. Ultimately, the powerful combination of automated processes and human expertise sets the stage for advanced technologies capable of producing data that meets the rigorous demands of training high-quality AI systems on a large scale.
Conclusion:
As previously stated, the federal government faces a huge challenge in gathering enough quality diverse training datasets suitable for AI production at scale, and manually labeling data isn’t practical when responding to real-time global threats. Leveraging Large Language Models (LLMs) offers a way forward, making it possible to enrich and expand our data more efficiently. Yet, this approach isn’t perfect—issues like biases, inconsistent quality, and the complexities of designing the right prompts require careful consideration. The best strategy to date? A smart mix of human expertise with LLM capabilities, enhanced by domain specific prompt engineering with incremental improvements against ‘ground truth’ benchmarks. This blend of technical and human insight seems to break through the data labeling bottleneck and potentially pave the way for AI to serve the broader public interest; turning data challenges into opportunities for innovation and progress.
REFERENCES:
- Automating Data Annotation with LLMs // LLMs in Production Conference 3 Workshop1, MLOps community, Nikolai Liubimov, Michael Malyuk, Chris Hoge, Oct 20, 2023 LLM in Production – Part III
- What Is A Large Language Model (LLM)? A Complete Guide, Aminu Abdullahi – February 15, 2024
- Automatic data labeling with LLMs, Anita Kirkovska, Nov 2, 2023
- OLMo: Everything You Need to Train an Open Source LLM, Akshita Bhagia, Twiml AI Podcast with Sam Charrington, Mar 04, 2024
- Know about Zero Shot, One Shot and Few Shot Learning, Sarojag, 04 Dec 2023
- How programmatic labeling can minimize data exposure, Devang Sachdev, December 21, 2022
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions, Lei Huang, 9 Nov 2023
