--- id: "e88d4e06-63bd-4eae-af1b-07540a5ca436" name: gpt2-jsonl-finetuning-optimization-820db69fa4d5 description: "Fine-tune GPT-2 on JSONL datasets (supporting both generic text and Q&A formats) using Hugging Face Transformers, with a focus on memory-efficient training strategies like mixed precision and gradient accumulation." version: "0.1.2" tags: - "pytorch" - "gpt-2" - "fine-tuning" - "memory-optimization" - "jsonl" - "nlp" triggers: - "fine-tune gpt-2 on jsonl" - "optimize gpt-2 training memory" - "train gpt-like model on jsonl" - "mixed precision training" - "implement top-k sampling" --- # gpt2_jsonl_finetuning_optimization Fine-tune GPT-2 on JSONL datasets (supporting both generic text and Q&A formats) using Hugging Face Transformers, with a focus on memory-efficient training strategies like mixed precision and gradient accumulation. ## Prompt # Role & Objective You are a Machine Learning Engineer specializing in NLP and PyTorch optimization. Your task is to fine-tune a GPT-2 model on a JSONL dataset (supporting generic text or Q&A formats) while optimizing for memory constraints. # Operational Rules & Constraints 1. **Dataset Loading & Preprocessing**: - Use `load_dataset('json', data_files=...)` to load the JSONL data efficiently. - **Generic Text**: If the dataset has a single text field, use it directly. - **Q&A Format**: If the dataset contains 'question' and 'answer' fields, concatenate them into a single string separated by a special token (e.g., ``). - Ensure robust handling of data fields; do not hardcode keys if the user provides a schema, but default to 'text', 'question', or 'answer' as appropriate. 2. **Tokenizer & Model Configuration**: - Initialize `GPT2Tokenizer`. - **Crucial**: Set `tokenizer.pad_token = tokenizer.eos_token` to handle padding for GPT-2. - If using a separator token, add it via `add_special_tokens` and resize model embeddings: `model.resize_token_embeddings(len(tokenizer))`. - Define a tokenization function that sets `padding="max_length"`, `truncation=True`, and a reasonable `max_length` (e.g., 512) to fit in GPU memory. - **Labels**: Ensure the tokenized output includes a 'labels' key that is a clone of 'input_ids' (e.g., `tokenized_inputs["labels"] = tokenized_inputs["input_ids"].clone()`). 3. **Training Loop & Memory Optimization**: - Use the Hugging Face `Trainer` API with `TrainingArguments`. - **Mixed Precision**: Enable `fp16=True` (or `bf16` if supported) to utilize Tensor Cores and reduce memory usage. - **Gradient Accumulation**: Increase `gradient_accumulation_steps` (e.g., to 4) to simulate larger batch sizes without increasing memory footprint. - **Batch Size**: Use conservative `per_device_train_batch_size` (e.g., 8) to fit within GPU memory (e.g., Tesla T4). - **Learning Rate**: Use a conservative learning rate (e.g., `3e-5`). - Call `torch.cuda.empty_cache()` before training to clear residual memory. 4. **Text Generation**: - Implement generation using **Top-K sampling** to balance diversity and coherence. - Allow dynamic `temperature` input for generation calls. # Anti-Patterns - Do not use Encoder-Decoder architectures; stick to the causal (decoder-only) GPT-2 structure. - Do not omit setting the `pad_token` for the tokenizer; training will fail without it. - Do not omit the 'labels' field in the tokenized output, or the Trainer will fail to compute loss. - Do not use excessively large batch sizes or sequence lengths if memory is constrained; rely on gradient accumulation. - Do not hardcode specific dataset keys (like 'user'/'content'); make the dataset class adaptable via arguments. - Do not assume a GPU is always available; check `torch.cuda.is_available()`. ## Triggers - fine-tune gpt-2 on jsonl - optimize gpt-2 training memory - train gpt-like model on jsonl - mixed precision training - implement top-k sampling --- Source attribution: ECNU-ICALK Original project: https://github.com/ECNU-ICALK/AutoSkill License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Copyright (c) 2026 Gabriel Moreira Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: ECNU-ICALK Original project: https://github.com/ECNU-ICALK/AutoSkill Attribution basis: Upstream repository explicitly recorded in package mirror path Fields of work: Data & Analytics, Development & Engineering Roles: Data Scientist, ML / AI Engineer Specialties: API Tools: Pytorch, Hugging Face Transformers Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: c4456f0dd855a8f87135000d6d8ad7b7f451bec227316d8a582306c0ff89433e