PJMixers-Archive/LLaMa-3-Instruct-ClownMoE-Prompts

A list of prompts to help with the creation of Clown MoEs. Pull requests/suggestions are encouraged, especially for rewriting current questions to be less repetitive as these are being made with ChatGPT first then will be reworded later.

If you are using the latest LLaMa-3-Instruct model's tokenizer these prompts should be formatted correctly. If you are using an outdated version you may need to add the BOS manually.

I've included YAML files for each category so you can copy to your mergekit config easier.

Example recipe

This one only uses positive prompts.

base_model: meta-llama/Meta-Llama-3-8B-Instruct
gate_mode: hidden
dtype: bfloat16
experts_per_token: 2
experts:
  - source_model: Some LLaMa-3-Instruct finetune that only does math good
    positive_prompts:
      - "<|start_header_id|>user<|end_header_id|>\n\nSolve for x in the equation: 3x + 5 = 14.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nWhat is the result of 56 divided by 7?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nCalculate the product of 23 and 4.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nIf a pizza is divided into 8 equal slices and 3 slices are eaten, how many slices are left?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
  - source_model: Some LLaMa-3-Instruct finetune that only does science good
    positive_prompts:
      - "<|start_header_id|>user<|end_header_id|>\n\nWhat is the chemical symbol for gold?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nWhat is the process by which plants use sunlight to produce glucose called?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nName the organelle responsible for producing energy in eukaryotic cells.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nDescribe the difference between mitosis and meiosis.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
  - source_model: Some LLaMa-3-Instruct finetune that only does geography good
    positive_prompts:
      - "<|start_header_id|>user<|end_header_id|>\n\nWhat's the capital of Canada?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nWhich city is the capital of Brazil?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nName the capital of South Africa.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nWhat is the longest river in the world?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
  - source_model: Some LLaMa-3-Instruct finetune that only does coding good
    positive_prompts:
      - "<|start_header_id|>user<|end_header_id|>\n\nImplement a function to calculate the factorial of a non-negative integer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nWrite a program to determine if a given string is a palindrome.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nCreate an algorithm to find the nth Fibonacci number.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nImplement a stack data structure using arrays or linked lists.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"

This one includes negative prompts for the other categories to emphasize that this expert shouldn't be chosen for this type of prompt.

base_model: meta-llama/Meta-Llama-3-8B-Instruct
gate_mode: hidden
dtype: bfloat16
experts_per_token: 2
experts:
  - source_model: Some LLaMa-3-Instruct finetune that only does math good
    positive_prompts:
      - "<|start_header_id|>user<|end_header_id|>\n\nSolve for x in the equation: 3x + 5 = 14.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nWhat is the result of 56 divided by 7?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nCalculate the product of 23 and 4.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nIf a pizza is divided into 8 equal slices and 3 slices are eaten, how many slices are left?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
    negative_prompts:
      - "<|start_header_id|>user<|end_header_id|>\n\nDefine the term \"acid\" and provide an example of a common acid.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nWhich mountain range spans across several countries in Europe?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nWrite a program to check if a binary tree is balanced.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
  - source_model: Some LLaMa-3-Instruct finetune that only does science good
    positive_prompts:
      - "<|start_header_id|>user<|end_header_id|>\n\nWhat is the chemical symbol for gold?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nWhat is the process by which plants use sunlight to produce glucose called?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nName the organelle responsible for producing energy in eukaryotic cells.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nDescribe the difference between mitosis and meiosis.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
    negative_prompts:
      - "<|start_header_id|>user<|end_header_id|>\n\nSolve for x: 2x + 5 = 15.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nName the largest desert in the world.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nCreate a function to reverse a linked list.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
- source_model: Some LLaMa-3-Instruct finetune that only does geography good
    positive_prompts:
      - "<|start_header_id|>user<|end_header_id|>\n\nWhat's the capital of Canada?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nWhich city is the capital of Brazil?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nName the capital of South Africa.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nWhat is the longest river in the world?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
    negative_prompts:
      - "<|start_header_id|>user<|end_header_id|>\n\nFactor the expression: x^2 - 4.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nWhat is the chemical formula for water?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nImplement the bubble sort algorithm.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
  - source_model: Some LLaMa-3-Instruct finetune that only does coding good
    positive_prompts:
      - "<|start_header_id|>user<|end_header_id|>\n\nImplement a function to calculate the factorial of a non-negative integer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nWrite a program to determine if a given string is a palindrome.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nCreate an algorithm to find the nth Fibonacci number.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nImplement a stack data structure using arrays or linked lists.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
    negative_prompts:
      - "<|start_header_id|>user<|end_header_id|>\n\nFind the value of y if 3y - 7 = 17.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nExplain the concept of electronegativity in chemical bonding.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
      - "<|start_header_id|>user<|end_header_id|>\n\nHow many countries are there in Africa?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"