# Guide to Huggingface – WizardCoder

## Introduction to WizardCoder: Empowering Code Large Language Models with Evol-Instruct
WizardCoder is a large language model developed by WizardLM, which has been trained using Evol-Instruct. This model has achieved state-of-the-art performance in the open-source code large language model (LLM) domain. If you are interested in exploring the capabilities and using the WizardCoder model, this guide will provide you with the necessary information to get started.

## Getting Started
To get started with WizardCoder, you can visit the following links:
– 🏠 [Home Page](https://wizardlm.github.io/)
– 🤗 [HuggingFace Repo](https://huggingface.co/WizardLM)
– 🐱 [Github Repo](https://github.com/nlpxucan/WizardLM)
– 🐦 [Twitter](https://twitter.com/WizardLM_AI)
– 📃 [WizardLM](https://arxiv.org/abs/2304.12244)
– 📃 [WizardCoder](https://arxiv.org/abs/2306.08568)
– 📃 [WizardMath](https://arxiv.org/abs/2308.09583)

You can also join the community on Discord: [Discord](https://discord.gg/VZjjHtWrKs)

## News
The latest release, WizardCoder-33B-V1.1, achieves excellent performance and outperforms other models on various evaluation benchmarks. Visit the [EvalPlus Leaderboard](https://evalplus.github.io/leaderboard.html) to see the performance metrics.

## How to Make the Training Data
To make the training data for WizardCoder, you can apply the Code Evol-Instruct on the provided Code-Alpaca dataset.

## Data Contamination Check
Before model training, it is essential to perform a rigorous data contamination check to ensure the integrity of the training data. Certain guidelines and prompts are provided for using the system prompts to maintain consistency and accuracy during the training process.

## How to Reproduce the Performance of WizardCoder-33B-V1.1
To reproduce the performance of WizardCoder-33B-V1.1, you can use the provided codes and generate the results. Detailed instructions and code snippets are provided for running the evaluation processes for HumanEval, HumanEval-Plus, MBPP, and MBPP-Plus.

## Citation
If you use the data, method, or code from this repository, please cite it using the following format:
“`
@article{luo2023wizardcoder,
title={WizardCoder: Empowering Code Large Language Models with Evol-Instruct},
author={Luo, Ziyang and Xu, Can and Zhao, Pu and Sun, Qingfeng and Geng, Xiubo and Hu, Wenxiang and Tao, Chongyang and Ma, Jing and Lin, Qingwei and Jiang, Daxin},
journal={arXiv preprint arXiv:2306.08568},
year={2023}
}
“`

With this guide, you have the essential information to explore and use the capabilities of HuggingFace – WizardCoder. Happy coding!

Source link
**Huggingface Tutorial: Empowering Code Large Language Models with Evol-Instruct**

Welcome to the Huggingface tutorial for using the WizardCoder model. In this tutorial, we will guide you through the process of utilizing the WizardCoder-33B-V1.1 model, which is empowered by the Evol-Instruct technique. This model is known for achieving state-of-the-art performance in generating code-based language models. Let’s get started!

**Getting Started with WizardCoder-33B-V1.1**

Before we begin, please make sure you have the necessary packages and frameworks installed.

**Requirements:**
– `transformers==4.36.2`
– `vllm==0.2.5`

**Usage**

1. **HumanEval and HumanEval-Plus**

– Code Generation (without acceleration)
“`bash
# Set model parameters
model=”WizardLM/WizardCoder-33B-V1.1″
temp=0.0
max_len=2048
pred_num=1
num_seqs_per_iter=1

# Generate code
python humaneval_gen.py –model ${model} \
–start_index 0 –end_index 20 –temperature ${temp} \
–num_seqs_per_iter ${num_seqs_per_iter} –N ${pred_num} –max_len ${max_len} –output_path preds/T${temp}_N${pred_num}_WizardCoder-33B-V1.1_Greedy_Decode
“`

– Code Generation (with vllm acceleration)
“`bash
model=”WizardLM/WizardCoder-33B-V1.1″
temp=0.0
max_len=2048
pred_num=1
num_seqs_per_iter=1

CUDA_VISIBLE_DEVICES=0,1,2,3 python humaneval_gen_vllm.py –model ${model} \
–start_index 0 –end_index 164 –temperature ${temp} \
–num_seqs_per_iter ${num_seqs_per_iter} –N ${pred_num} –max_len ${max_len} –output_path preds/T${temp}_N${pred_num}_WizardCoder-33B-V1.1_Greedy_Decode_vllm –num_gpus 4 –overwrite
“`

2. **MBPP and MBPP-Plus**

– Code Generation (without acceleration)
“`bash
model=”WizardLM/WizardCoder-33B-V1.1″
temp=0.0
max_len=2048
pred_num=1
num_seqs_per_iter=1

# Generate code
python mbppplus_gen.py –model ${model} \
–start_index 0 –end_index 50 –temperature ${temp} \
–num_seqs_per_iter ${num_seqs_per_iter} –N ${pred_num} –max_len ${max_len} –output_path preds/MBPP_T${temp}_N${pred_num}_WizardCoder-33B-V1.1_Greedy_Decode –mbpp_path “mbppplus.json” –greedy_decode
“`

– Code Generation (with vllm acceleration)
“`bash
model=”WizardLM/WizardCoder-33B-V1.1″
temp=0.0
max_len=2048
pred_num=1
num_seqs_per_iter=1

CUDA_VISIBLE_DEVICES=0,1,2,3 python mbppplus_gen_vllm.py –model ${model} \
–start_index 0 –end_index 50 –temperature ${temp} \
–num_seqs_per_iter ${num_seqs_per_iter} –N ${pred_num} –max_len ${max_len} –output_path preds/MBPP_T${temp}_N${pred_num}_WizardCoder-33B-V1.1_Greedy_Decode_vllm –mbpp_path “mbppplus.json” –num_gpus 4
“`

**Citation**

If you use the data, method, or code from this repository, please cite the following:

“`
@article{luo2023wizardcoder,
title={WizardCoder: Empowering Code Large Language Models with Evol-Instruct},
author={Luo, Ziyang and Xu, Can and Zhao, Pu and Sun, Qingfeng and Geng, Xiubo and Hu, Wenxiang and Tao, Chongyang and Ma, Jing and Lin, Qingwei and Jiang, Daxin},
journal={arXiv preprint arXiv:2306.08568},
year={2023}
}
“`

For any further assistance or queries, feel free to reach out to us on our [Discord](https://discord.gg/VZjjHtWrKs) channel.

Thank you for choosing Huggingface and WizardCoder for your code language modeling needs! We hope this tutorial helps you make the most of our powerful model.



WizardCoder: Empowering Code Large Language Models with Evol-Instruct

🏠 Home Page

🤗 HF Repo •🐱 Github Repo • 🐦 Twitter

📃 [WizardLM] • 📃 [WizardCoder] • 📃 [WizardMath]

👋 Join our Discord



News

[2024/01/04] 🔥 We released WizardCoder-33B-V1.1 trained from deepseek-coder-33b-base, the SOTA OSS Code LLM on EvalPlus Leaderboard, achieves 79.9 pass@1 on HumanEval, 73.2 pass@1 on HumanEval-Plus, 78.9 pass@1 on MBPP, and 66.9 pass@1 on MBPP-Plus.

[2024/01/04] 🔥 WizardCoder-33B-V1.1 outperforms ChatGPT 3.5, Gemini Pro, and DeepSeek-Coder-33B-instruct on HumanEval and HumanEval-Plus pass@1.

[2024/01/04] 🔥 WizardCoder-33B-V1.1 is comparable with ChatGPT 3.5, and surpasses Gemini Pro on MBPP and MBPP-Plus pass@1.



How to Make the Training Data?

Apply our Code Evol-Instruct on Code-Aplaca data.



❗ Data Contamination Check:

Before model training, we carefully and rigorously checked all the training data, and used multiple deduplication methods to verify and prevent data leakage on HumanEval and MBPP test set.

🔥
Note for model system prompts usage:

Please use the same systems prompts strictly with us, and we do not guarantee the accuracy of the quantified versions.

Default version:

"Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Response:"



How to Reproduce the Performance of WizardCoder-33B-V1.1

We provide all codes here.

We also provide all generated results.

transformers==4.36.2
vllm==0.2.5

(1) HumanEval and HumanEval-Plus

Code Generation (w/o accelerate)

model="WizardLM/WizardCoder-33B-V1.1"
temp=0.0
max_len=2048
pred_num=1
num_seqs_per_iter=1

output_path=preds/T${temp}_N${pred_num}_WizardCoder-33B-V1.1_Greedy_Decode

mkdir -p ${output_path}
echo 'Output path: '$output_path
echo 'Model to eval: '$model


index=0
gpu_num=8
for ((i = 0; i < $gpu_num; i++)); do
  start_index=$((i * 21))
  end_index=$(((i + 1) * 21))

  gpu=$((i))
  echo 'Running process #' ${i} 'from' $start_index 'to' $end_index 'on GPU' ${gpu}
  ((index++))
  (
    CUDA_VISIBLE_DEVICES=$gpu python humaneval_gen.py --model ${model} \
      --start_index ${start_index} --end_index ${end_index} --temperature ${temp} \
      --num_seqs_per_iter ${num_seqs_per_iter} --N ${pred_num} --max_len ${max_len} --output_path ${output_path} --greedy_decode
  ) &
  if (($index % $gpu_num == 0)); then wait; fi
done

Code Generation (w/ vllm accelerate)

model="WizardLM/WizardCoder-33B-V1.1"
temp=0.0
max_len=2048
pred_num=1
num_seqs_per_iter=1

output_path=preds/T${temp}_N${pred_num}_WizardCoder-33B-V1.1_Greedy_Decode_vllm

mkdir -p ${output_path}
echo 'Output path: '$output_path
echo 'Model to eval: '$model

CUDA_VISIBLE_DEVICES=0,1,2,3 python humaneval_gen_vllm.py --model ${model} \
    --start_index 0 --end_index 164 --temperature ${temp} \
    --num_seqs_per_iter ${num_seqs_per_iter} --N ${pred_num} --max_len ${max_len} --output_path ${output_path} --num_gpus 4 --overwrite

Install Eval-Plus benchmark.

git clone https://github.com/evalplus/evalplus.git
cd evalplus
export PYTHONPATH=$PYTHONPATH:$(pwd)
pip install -r requirements.txt

Get HumanEval and HumanEval-Plus scores.

output_path=preds/T0.0_N1_WizardCoder-33B-V1.1_Greedy_Decode

echo 'Output path: '$output_path
python process_humaneval.py --path ${output_path} --out_path ${output_path}.jsonl --add_prompt

evalplus.evaluate --dataset humaneval --samples ${output_path}.jsonl

(2) MBPP and MBPP-Plus

The preprocessed questions are provided in mbppplus.json.

Code Generation (w/o accelerate)

model="WizardLM/WizardCoder-33B-V1.1"
temp=0.0
max_len=2048
pred_num=1
num_seqs_per_iter=1

output_path=preds/MBPP_T${temp}_N${pred_num}_WizardCoder-33B-V1.1_Greedy_Decode

mkdir -p ${output_path}
echo 'Output path: '$output_path
echo 'Model to eval: '$model


index=0
gpu_num=8
for ((i = 0; i < $gpu_num; i++)); do
  start_index=$((i * 50))
  end_index=$(((i + 1) * 50))

  gpu=$((i))
  echo 'Running process #' ${i} 'from' $start_index 'to' $end_index 'on GPU' ${gpu}
  ((index++))
  (
    CUDA_VISIBLE_DEVICES=$gpu python mbppplus_gen.py --model ${model} \
      --start_index ${start_index} --end_index ${end_index} --temperature ${temp} \
      --num_seqs_per_iter ${num_seqs_per_iter} --N ${pred_num} --max_len ${max_len} --output_path ${output_path} --mbpp_path "mbppplus.json" --greedy_decode
  ) &
  if (($index % $gpu_num == 0)); then wait; fi
done

Code Generation (w/ vllm accelerate)

model="WizardLM/WizardCoder-33B-V1.1"
temp=0.0
max_len=2048
pred_num=1
num_seqs_per_iter=1

output_path=preds/MBPP_T${temp}_N${pred_num}_WizardCoder-33B-V1.1_Greedy_Decode_vllm

mkdir -p ${output_path}
echo 'Output path: '$output_path
echo 'Model to eval: '$model

CUDA_VISIBLE_DEVICES=0,1,2,3 python mbppplus_gen_vllm.py --model ${model} \
    --start_index ${start_index} --end_index ${end_index} --temperature ${temp} \
    --num_seqs_per_iter ${num_seqs_per_iter} --N ${pred_num} --max_len ${max_len} --output_path ${output_path} --mbpp_path "mbppplus.json" --num_gpus 4

Install Eval-Plus benchmark.

git clone https://github.com/evalplus/evalplus.git
cd evalplus
export PYTHONPATH=$PYTHONPATH:$(pwd)
pip install -r requirements.txt

Get HumanEval and HumanEval-Plus scores.

output_path=preds/MBPP_T0.0_N1_WizardCoder-33B-V1.1_Greedy_Decode

echo 'Output path: '$output_path
python mbppplus_process_preds.py --path ${output_path} --out_path ${output_path}.jsonl --add_prompt

evalplus.evaluate --dataset mbpp --samples ${output_path}.jsonl



Citation

Please cite the repo if you use the data, method or code in this repo.

@article{luo2023wizardcoder,
  title={WizardCoder: Empowering Code Large Language Models with Evol-Instruct},
  author={Luo, Ziyang and Xu, Can and Zhao, Pu and Sun, Qingfeng and Geng, Xiubo and Hu, Wenxiang and Tao, Chongyang and Ma, Jing and Lin, Qingwei and Jiang, Daxin},
  journal={arXiv preprint arXiv:2306.08568},
  year={2023}
}


The HTML code provided contains multiple use cases where the

tag is utilized. Let’s break down the use cases below:

1. Navigation: The

tag contains navigation links to various resources related to the WizardCoder project, such as the Home Page, HF Repo, Github Repo, Twitter, and various arXiv links. This allows users to easily navigate to different web pages related to the project.

2. News Section: The

tag contains recent news updates about the WizardCoder project. These updates include information about the release of the WizardCoder-33B-V1.1 model, its performance on different evaluation metrics, and its comparison with other code language models. This section provides users with the latest news and updates about the project.

3. How to Make the Training Data: The

tag includes a link to the Code Evol-Instruct application and provides guidance on how to apply it to the Code-Alpaca dataset. This section is instructional, providing users with information on how to generate training data for the code language model.

4. Data Contamination Check: The

tag contains important information about the data contamination check process. It emphasizes the careful verification and deduplication of training data to prevent data leakage, ensuring the integrity of the model’s training data.

5. How to Reproduce the Performance of WizardCoder-33B-V1.1: The

tag provides users with links to access the code and results for reproducing the performance of the WizardCoder-33B-V1.1 model. It includes detailed instructions and code snippets for running evaluations on different datasets and generating performance results.

6. Citation: The

tag contains a request for users to cite the WizardCoder repository if they use the data, method, or code in their work. It includes a formatted citation for the WizardCoder project, making it easy for users to properly acknowledge and cite the project in their research.

Overall, the

tag is used to structure and organize the diverse content and information related to the WizardCoder project. It plays a crucial role in presenting navigation links, news updates, instructional content, code snippets, and citation requests to the users. The structured use of the

tag enhances user experience and facilitates access to the various resources and information provided on the webpage.

2024-01-09T03:37:59+01:00