longeval/topics_lines_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Turotial: Coarse-grained Topic Retrieval and Fine-grained Line Retrieval Evaluation ###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we present the process of evaluating the model LongChat-13B-16K on Topic Retrieval with 5-topic testcases and use our auto_topic_eval module to check the accuracy of the outputs. We demonstrate how to run the Line Retrieval evaluation with 200-line testcases as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `eval.py` for the Topic and the Line Retrieval evaluation. \n",
    "\n",
    "* `--model-name-or-path`: huggingface model name or path to the checkpoint.\n",
    "* `--task`: pass \"topics\" for Topic Retrieval evaluation; pass \"lines\" for Line Retrieval evaluation.\n",
    "* `--longchat_flash_attn`: enable flash attention that only applies to the LongChat models\n",
    "* `--eval_shortest_only`: evaluate only on the shortest context lenght. Remove it to reproduce our full table.\n",
    "* `--num_gpus`: number of GPGs to be used for evaluation. Default is 1.\n",
    "* `--max_gpu_memory`: maximum GPU memory (GiB) can be used per GPU to load the model. Default is 40.\n",
    "\n",
    "To check the final accuracy: \n",
    "\n",
    "* No extra action needed for Line Retrieval. The accuracy is automaticly evaluated and printed in the logs.\n",
    "* We use `auto_topic_eval.py` to evaluate Topic Retrieval using ChatGPT.\n",
    "    * `--test_file`: the output file generated by eval.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Topic Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output to evaluation/topics/predictions/longchat-13b-16k\n",
      "lmsys/longchat-13b-16k\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Loading checkpoint shards: 100%|██████████████████| 3/3 [00:10<00:00,  3.40s/it]\n",
      "************ Start testing 5 topics per prompt ***********\n",
      "0it [00:00, ?it/s]Label: The psychology of creativity, Predict: ['The first topic we discussed was the psychology of creativity.'], prompt length: 3430\n",
      "1it [00:11, 11.75s/it]Label: The benefits of learning a new language, Predict: ['The first topic we discussed was the benefits of learning a new language.'], prompt length: 3297\n",
      "2it [00:19,  9.68s/it]Label: The effects of climate change on ocean ecosystems, Predict: ['The first topic we discussed was the effects of climate change on ocean ecosystems.'], prompt length: 3333\n",
      "3it [00:30, 10.18s/it]Label: The role of art in society, Predict: ['The first topic we discussed was the role of art in society.'], prompt length: 3290\n",
      "4it [00:38,  9.21s/it]Label: The effects of climate change on ocean ecosystems, Predict: ['The first topic we discussed was the effects of climate change on ocean ecosystems.'], prompt length: 3348\n",
      "5it [00:49,  9.81s/it]Label: The benefits of volunteering, Predict: ['The first topic we discussed was the benefits of volunteering.'], prompt length: 3088\n",
      "6it [00:56,  8.85s/it]Label: The history and impact of the Renaissance, Predict: ['The first topic we discussed was the history and impact of the Renaissance.'], prompt length: 3367\n",
      "7it [01:04,  8.78s/it]Label: The role of sports in society, Predict: ['The first topic we discussed was the role of sports in society.'], prompt length: 3171\n",
      "8it [01:12,  8.39s/it]Label: The psychology of happiness, Predict: ['The first topic we discussed was the psychology of happiness.'], prompt length: 3095\n",
      "9it [01:19,  7.95s/it]Label: The history and culture of ancient civilizations, Predict: ['The first topic we discussed was the history and culture of ancient civilizations.'], prompt length: 3213\n",
      "10it [01:28,  8.19s/it]Label: The effects of stress on the body and mind, Predict: ['The first topic we discussed was the effects of stress on the body and mind.'], prompt length: 3475\n",
      "11it [01:39,  9.04s/it]Label: The benefits of spending time in nature, Predict: ['The first topic we discussed was the benefits of spending time in nature.'], prompt length: 3461\n",
      "12it [01:48,  9.25s/it]Label: The psychology of creativity, Predict: ['The first topic we discussed was the psychology of creativity.'], prompt length: 3535\n",
      "13it [01:57,  9.06s/it]Label: The benefits of regular exercise, Predict: ['The first topic we discussed was the benefits of regular exercise.'], prompt length: 3185\n",
      "14it [02:04,  8.44s/it]Label: The role of education in society, Predict: ['The first topic we discussed was \"The role of education in society.\"'], prompt length: 3405\n",
      "15it [02:13,  8.49s/it]Label: The effects of sleep on overall health, Predict: ['The first topic we discussed was the effects of sleep on overall health.'], prompt length: 3253\n",
      "16it [02:21,  8.41s/it]Label: The psychology of happiness, Predict: ['The first topic we discussed was the psychology of happiness.'], prompt length: 3315\n",
      "17it [02:28,  8.04s/it]Label: The effects of climate change on ocean ecosystems, Predict: ['The first topic we discussed was the effects of climate change on ocean ecosystems.'], prompt length: 3081\n",
      "18it [02:38,  8.69s/it]Label: The future of sustainable agriculture, Predict: ['The first topic we discussed was the future of sustainable agriculture.'], prompt length: 3178\n",
      "19it [02:47,  8.67s/it]Label: The future of sustainable agriculture, Predict: ['The first topic we discussed was the future of sustainable agriculture.'], prompt length: 3386\n",
      "20it [02:56,  8.82s/it]Label: The benefits of regular exercise, Predict: ['The first topic we discussed was the benefits of regular exercise.'], prompt length: 3163\n",
      "21it [03:03,  8.28s/it]Label: The role of sports in society, Predict: ['The first topic we discussed was the role of sports in society.'], prompt length: 3247\n",
      "22it [03:11,  8.09s/it]Label: The history and culture of ancient civilizations, Predict: ['The first topic we discussed was the history and culture of ancient civilizations.'], prompt length: 3138\n",
      "23it [03:19,  8.24s/it]Label: The role of sports in society, Predict: ['The first topic we discussed was the role of sports in society.'], prompt length: 2972\n",
      "24it [03:26,  7.90s/it]Label: The role of art in society, Predict: ['The first topic we discussed was the role of art in society.'], prompt length: 3402\n",
      "25it [03:34,  7.94s/it]Label: The effects of sleep on overall health, Predict: ['The first topic we discussed was the effects of sleep on overall health.'], prompt length: 3114\n",
      "26it [03:43,  7.98s/it]Label: The impact of technology on human connection, Predict: ['The first topic we discussed was the impact of technology on human connection.'], prompt length: 3160\n",
      "27it [03:51,  8.01s/it]Label: The future of sustainable agriculture, Predict: ['The first topic we discussed was the future of sustainable agriculture.'], prompt length: 3223\n",
      "28it [03:59,  8.22s/it]Label: The role of art in society, Predict: ['The first topic we discussed was the role of art in society.'], prompt length: 3238\n",
      "29it [04:07,  8.05s/it]Label: The benefits of regular exercise, Predict: ['The first topic we discussed was the benefits of regular exercise.'], prompt length: 3308\n",
      "30it [04:14,  7.79s/it]Label: The effects of stress on the body and mind, Predict: ['The first topic we discussed was the effects of stress on the body and mind.'], prompt length: 3269\n",
      "31it [04:23,  8.25s/it]Label: The effects of climate change on ocean ecosystems, Predict: ['The first topic we discussed was the effects of climate change on ocean ecosystems.'], prompt length: 3324\n",
      "32it [04:34,  8.99s/it]Label: The benefits of regular exercise, Predict: ['The first topic we discussed was the benefits of regular exercise.'], prompt length: 3294\n",
      "33it [04:41,  8.44s/it]Label: The effects of sleep on overall health, Predict: ['The first topic we discussed was the effects of sleep on overall health.'], prompt length: 3159\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "34it [04:49,  8.33s/it]Label: The role of sports in society, Predict: ['The first topic we discussed was the role of sports in society.'], prompt length: 3351\n",
      "35it [04:57,  8.24s/it]Label: The benefits of reading for pleasure, Predict: ['The first topic we discussed was the benefits of reading for pleasure.'], prompt length: 3559\n",
      "36it [05:06,  8.37s/it]Label: The role of sports in society, Predict: ['The first topic we discussed was the role of sports in society.'], prompt length: 3069\n",
      "37it [05:14,  8.08s/it]Label: The effects of air pollution on human health, Predict: ['The first topic we discussed was the effects of air pollution on human health.'], prompt length: 3096\n",
      "38it [05:23,  8.40s/it]Label: The role of art in society, Predict: ['The first topic we discussed was the role of art in society.'], prompt length: 3398\n",
      "39it [05:31,  8.28s/it]Label: The impact of social media on communication, Predict: ['The first topic we discussed was the impact of social media on communication.'], prompt length: 3490\n",
      "40it [05:40,  8.54s/it]Label: The psychology of creativity, Predict: ['The first topic we discussed was the psychology of creativity.'], prompt length: 3304\n",
      "41it [05:48,  8.30s/it]Label: The effects of stress on the body and mind, Predict: ['The first topic we discussed was the effects of stress on the body and mind.'], prompt length: 3535\n",
      "42it [05:58,  8.95s/it]Label: The psychology of creativity, Predict: ['The first topic we discussed was the psychology of creativity.'], prompt length: 3182\n",
      "43it [06:06,  8.53s/it]Label: The future of renewable energy storage, Predict: ['The first topic we discussed was the future of renewable energy storage.'], prompt length: 3333\n",
      "44it [06:14,  8.53s/it]Label: The benefits of a plant-based diet, Predict: ['The first topic we discussed was the benefits of a plant-based diet.'], prompt length: 3304\n",
      "45it [06:24,  8.79s/it]Label: The benefits of regular exercise, Predict: ['The first topic we discussed was the benefits of regular exercise.'], prompt length: 3389\n",
      "46it [06:31,  8.38s/it]Label: The future of sustainable agriculture, Predict: ['The first topic we discussed was the future of sustainable agriculture.'], prompt length: 3060\n",
      "47it [06:39,  8.36s/it]Label: The role of sports in society, Predict: ['The first topic we discussed was the role of sports in society.'], prompt length: 3103\n",
      "48it [06:47,  8.11s/it]Label: The impact of social media on communication, Predict: ['The first topic we discussed was the impact of social media on communication.'], prompt length: 3464\n",
      "49it [06:56,  8.42s/it]Label: The role of education in society, Predict: ['The first topic we discussed was \"The role of education in society.\"'], prompt length: 3221\n",
      "50it [07:04,  8.49s/it]\n",
      "************ Finish testing 5 topics per prompt with average prompt length 3275.5 ************\n"
     ]
    }
   ],
   "source": [
    "!python3 eval.py --model-name-or-path  lmsys/longchat-13b-16k --task topics --longchat_flash_attn --eval_shortest_only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------- Start auto-evaluation, you should verify it does this correctly --------------\n",
      "Question #0: Label: The psychology of creativity, Predict: 'The first topic we discussed was the psychology of creativity. - auto-eval goes with correct\n",
      "Question #1: Label: The benefits of learning a new language, Predict: 'The first topic we discussed was the benefits of learning a new language. - auto-eval goes with correct\n",
      "Question #2: Label: The effects of climate change on ocean ecosystems, Predict: 'The first topic we discussed was the effects of climate change on ocean ecosystems. - auto-eval goes with correct\n",
      "Question #3: Label: The role of art in society, Predict: 'The first topic we discussed was the role of art in society. - auto-eval goes with correct\n",
      "Question #4: Label: The effects of climate change on ocean ecosystems, Predict: 'The first topic we discussed was the effects of climate change on ocean ecosystems. - auto-eval goes with correct\n",
      "Question #5: Label: The benefits of volunteering, Predict: 'The first topic we discussed was the benefits of volunteering. - auto-eval goes with correct\n",
      "Question #6: Label: The history and impact of the Renaissance, Predict: 'The first topic we discussed was the history and impact of the Renaissance. - auto-eval goes with correct\n",
      "Question #7: Label: The role of sports in society, Predict: 'The first topic we discussed was the role of sports in society. - auto-eval goes with correct\n",
      "Question #8: Label: The psychology of happiness, Predict: 'The first topic we discussed was the psychology of happiness. - auto-eval goes with correct\n",
      "Question #9: Label: The history and culture of ancient civilizations, Predict: 'The first topic we discussed was the history and culture of ancient civilizations. - auto-eval goes with correct\n",
      "Question #10: Label: The effects of stress on the body and mind, Predict: 'The first topic we discussed was the effects of stress on the body and mind. - auto-eval goes with correct\n",
      "The server is overloaded or not ready yet.\n",
      "Question #11: Label: The benefits of spending time in nature, Predict: 'The first topic we discussed was the benefits of spending time in nature. - auto-eval goes with correct\n",
      "Question #12: Label: The psychology of creativity, Predict: 'The first topic we discussed was the psychology of creativity. - auto-eval goes with correct\n",
      "Question #13: Label: The benefits of regular exercise, Predict: 'The first topic we discussed was the benefits of regular exercise. - auto-eval goes with correct\n",
      "Question #14: Label: The role of education in society, Predict: 'The first topic we discussed was \"The role of education in society.\" - auto-eval goes with correct\n",
      "Question #15: Label: The effects of sleep on overall health, Predict: 'The first topic we discussed was the effects of sleep on overall health. - auto-eval goes with correct\n",
      "Question #16: Label: The psychology of happiness, Predict: 'The first topic we discussed was the psychology of happiness. - auto-eval goes with correct\n",
      "Question #17: Label: The effects of climate change on ocean ecosystems, Predict: 'The first topic we discussed was the effects of climate change on ocean ecosystems. - auto-eval goes with correct\n",
      "Question #18: Label: The future of sustainable agriculture, Predict: 'The first topic we discussed was the future of sustainable agriculture. - auto-eval goes with correct\n",
      "Question #19: Label: The future of sustainable agriculture, Predict: 'The first topic we discussed was the future of sustainable agriculture. - auto-eval goes with correct\n",
      "Question #20: Label: The benefits of regular exercise, Predict: 'The first topic we discussed was the benefits of regular exercise. - auto-eval goes with correct\n",
      "Question #21: Label: The role of sports in society, Predict: 'The first topic we discussed was the role of sports in society. - auto-eval goes with correct\n",
      "Question #22: Label: The history and culture of ancient civilizations, Predict: 'The first topic we discussed was the history and culture of ancient civilizations. - auto-eval goes with correct\n",
      "Question #23: Label: The role of sports in society, Predict: 'The first topic we discussed was the role of sports in society. - auto-eval goes with correct\n",
      "Question #24: Label: The role of art in society, Predict: 'The first topic we discussed was the role of art in society. - auto-eval goes with correct\n",
      "Question #25: Label: The effects of sleep on overall health, Predict: 'The first topic we discussed was the effects of sleep on overall health. - auto-eval goes with correct\n",
      "Question #26: Label: The impact of technology on human connection, Predict: 'The first topic we discussed was the impact of technology on human connection. - auto-eval goes with correct\n",
      "Question #27: Label: The future of sustainable agriculture, Predict: 'The first topic we discussed was the future of sustainable agriculture. - auto-eval goes with correct\n",
      "Question #28: Label: The role of art in society, Predict: 'The first topic we discussed was the role of art in society. - auto-eval goes with correct\n",
      "Question #29: Label: The benefits of regular exercise, Predict: 'The first topic we discussed was the benefits of regular exercise. - auto-eval goes with correct\n",
      "Question #30: Label: The effects of stress on the body and mind, Predict: 'The first topic we discussed was the effects of stress on the body and mind. - auto-eval goes with correct\n",
      "Question #31: Label: The effects of climate change on ocean ecosystems, Predict: 'The first topic we discussed was the effects of climate change on ocean ecosystems. - auto-eval goes with correct\n",
      "Question #32: Label: The benefits of regular exercise, Predict: 'The first topic we discussed was the benefits of regular exercise. - auto-eval goes with correct\n",
      "Question #33: Label: The effects of sleep on overall health, Predict: 'The first topic we discussed was the effects of sleep on overall health. - auto-eval goes with correct\n",
      "Question #34: Label: The role of sports in society, Predict: 'The first topic we discussed was the role of sports in society. - auto-eval goes with correct\n",
      "Question #35: Label: The benefits of reading for pleasure, Predict: 'The first topic we discussed was the benefits of reading for pleasure. - auto-eval goes with correct\n",
      "Question #36: Label: The role of sports in society, Predict: 'The first topic we discussed was the role of sports in society. - auto-eval goes with correct\n",
      "Question #37: Label: The effects of air pollution on human health, Predict: 'The first topic we discussed was the effects of air pollution on human health. - auto-eval goes with correct\n",
      "Question #38: Label: The role of art in society, Predict: 'The first topic we discussed was the role of art in society. - auto-eval goes with correct\n",
      "Question #39: Label: The impact of social media on communication, Predict: 'The first topic we discussed was the impact of social media on communication. - auto-eval goes with correct\n",
      "Question #40: Label: The psychology of creativity, Predict: 'The first topic we discussed was the psychology of creativity. - auto-eval goes with correct\n",
      "Question #41: Label: The effects of stress on the body and mind, Predict: 'The first topic we discussed was the effects of stress on the body and mind. - auto-eval goes with correct\n",
      "Question #42: Label: The psychology of creativity, Predict: 'The first topic we discussed was the psychology of creativity. - auto-eval goes with correct\n",
      "Question #43: Label: The future of renewable energy storage, Predict: 'The first topic we discussed was the future of renewable energy storage. - auto-eval goes with correct\n",
      "Question #44: Label: The benefits of a plant-based diet, Predict: 'The first topic we discussed was the benefits of a plant-based diet. - auto-eval goes with correct\n",
      "Question #45: Label: The benefits of regular exercise, Predict: 'The first topic we discussed was the benefits of regular exercise. - auto-eval goes with correct\n",
      "Question #46: Label: The future of sustainable agriculture, Predict: 'The first topic we discussed was the future of sustainable agriculture. - auto-eval goes with correct\n",
      "Question #47: Label: The role of sports in society, Predict: 'The first topic we discussed was the role of sports in society. - auto-eval goes with correct\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question #48: Label: The impact of social media on communication, Predict: 'The first topic we discussed was the impact of social media on communication. - auto-eval goes with correct\n",
      "Question #49: Label: The role of education in society, Predict: 'The first topic we discussed was \"The role of education in society.\" - auto-eval goes with correct\n",
      "---------- End auto-evaluation, predict accuracy 1.0 ---------------\n"
     ]
    }
   ],
   "source": [
    "!export OPENAI_API_KEY=<YOUR_API_KEY>\n",
    "!python3 auto_topic_eval.py --test_file evaluation/topics/predictions/longchat-13b-16k/5_response.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Line Retrieval Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output to evaluation/lines/predictions/longchat-13b-16k\n",
      "lmsys/longchat-13b-16k\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Condensing Positional embeddings from 16384 to 2048\n",
      "Loading checkpoint shards: 100%|██████████████████| 3/3 [00:10<00:00,  3.39s/it]\n",
      "************ Start testing 200 lines per LRT prompt ************\n",
      "0it [00:00, ?it/s]Using conversation template: vicuna_v1.1\n",
      "Label: 2416, Predict: The <REGISTER_CONTENT> in line torpid-kid is <2416>., Parsed: 2416, prompt length: 4803\n",
      "1it [00:23, 23.92s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 41869, Predict: The <REGISTER_CONTENT> in line moaning-conversation is <10207>., Parsed: 10207, prompt length: 4800\n",
      "2it [00:47, 23.49s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 14564, Predict: The <REGISTER_CONTENT> in line tacit-colonial is <14564>., Parsed: 14564, prompt length: 4812\n",
      "3it [01:09, 23.00s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 42229, Predict: The <REGISTER_CONTENT> in line rambunctious-formamide is <42229>., Parsed: 42229, prompt length: 4781\n",
      "4it [01:33, 23.41s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 41019, Predict: The <REGISTER_CONTENT> in line jolly-skean is <41019>., Parsed: 41019, prompt length: 4818\n",
      "5it [01:55, 22.75s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 40933, Predict: The <REGISTER_CONTENT> in line clumsy-piglet is <40933>., Parsed: 40933, prompt length: 4799\n",
      "6it [02:18, 22.92s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 2874, Predict: The <REGISTER_CONTENT> in line goofy-accent is <2874>., Parsed: 2874, prompt length: 4797\n",
      "7it [02:39, 22.47s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 12602, Predict: The <REGISTER_CONTENT> in line demonic-attack is <12602>., Parsed: 12602, prompt length: 4777\n",
      "8it [03:01, 22.15s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 45710, Predict: The <REGISTER_CONTENT> in line addicted-website is <45710>., Parsed: 45710, prompt length: 4864\n",
      "9it [03:23, 22.02s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 25551, Predict: The <REGISTER_CONTENT> in line aggressive-authentication is <25551>., Parsed: 25551, prompt length: 4838\n",
      "10it [03:44, 21.89s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 36337, Predict: The <REGISTER_CONTENT> in line aback-house is <36337>., Parsed: 36337, prompt length: 4806\n",
      "11it [04:05, 21.52s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 4676, Predict: The <REGISTER_CONTENT> in line slimy-connotation is <4676>., Parsed: 4676, prompt length: 4827\n",
      "12it [04:27, 21.55s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 40004, Predict: The <REGISTER_CONTENT> in line glossy-establishment is <40004>., Parsed: 40004, prompt length: 4798\n",
      "13it [04:51, 22.32s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 35101, Predict: The <REGISTER_CONTENT> in line wary-tenement is <35101>., Parsed: 35101, prompt length: 4826\n",
      "14it [05:12, 22.10s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 23765, Predict: The <REGISTER_CONTENT> in line short-witch is <23765>., Parsed: 23765, prompt length: 4808\n",
      "15it [05:33, 21.68s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 16484, Predict: The <REGISTER_CONTENT> in line comfortable-camel is <16484>., Parsed: 16484, prompt length: 4824\n",
      "16it [05:54, 21.39s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 35908, Predict: The <REGISTER_CONTENT> in line smoggy-unity is <12064>., Parsed: 12064, prompt length: 4807\n",
      "17it [06:15, 21.44s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 36087, Predict: The <REGISTER_CONTENT> in line faded-team is <36087>., Parsed: 36087, prompt length: 4793\n",
      "18it [06:36, 21.20s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 2702, Predict: The <REGISTER_CONTENT> in line daffy-career is <2702>., Parsed: 2702, prompt length: 4813\n",
      "19it [06:57, 21.31s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 1777, Predict: The <REGISTER_CONTENT> in line evasive-membership is <1777>., Parsed: 1777, prompt length: 4812\n",
      "20it [07:20, 21.64s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 31290, Predict: The <REGISTER_CONTENT> in line wild-somewhere is <31290>., Parsed: 31290, prompt length: 4809\n",
      "21it [07:41, 21.36s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 11530, Predict: The <REGISTER_CONTENT> in line dry-illiteracy is <11530>., Parsed: 11530, prompt length: 4832\n",
      "22it [08:02, 21.44s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 21575, Predict: The <REGISTER_CONTENT> in line thundering-modem is <21575>., Parsed: 21575, prompt length: 4797\n",
      "23it [08:25, 21.72s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 9771, Predict: The <REGISTER_CONTENT> in line drab-cyclone is <9771>., Parsed: 9771, prompt length: 4804\n",
      "24it [08:45, 21.41s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 839, Predict: The <REGISTER_CONTENT> in line cloudy-counsel is <839>., Parsed: 839, prompt length: 4804\n",
      "25it [09:06, 21.19s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 29079, Predict: The <REGISTER_CONTENT> in line understood-top-hat is <29079>., Parsed: 29079, prompt length: 4772\n",
      "26it [09:27, 21.27s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 1836, Predict: The <REGISTER_CONTENT> in line disturbed-mastication is <1836>., Parsed: 1836, prompt length: 4814\n",
      "27it [09:49, 21.36s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 14486, Predict: The <REGISTER_CONTENT> in line meek-amendment is <14486>., Parsed: 14486, prompt length: 4795\n",
      "28it [10:11, 21.66s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 31934, Predict: The <REGISTER_CONTENT> in line standing-shareholder is <31934>., Parsed: 31934, prompt length: 4796\n",
      "29it [10:32, 21.36s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 47936, Predict: The <REGISTER_CONTENT> in line merciful-bungalow is <47936>., Parsed: 47936, prompt length: 4803\n",
      "30it [10:55, 21.93s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 36145, Predict: The <REGISTER_CONTENT> in line fabulous-acquisition is <36145>., Parsed: 36145, prompt length: 4805\n",
      "31it [11:18, 22.07s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 30114, Predict: The <REGISTER_CONTENT> in line heavenly-deed is <30114>., Parsed: 30114, prompt length: 4789\n",
      "32it [11:39, 21.90s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 32851, Predict: The <REGISTER_CONTENT> in line dangerous-direction is <32851>., Parsed: 32851, prompt length: 4794\n",
      "33it [11:59, 21.26s/it]Using conversation template: vicuna_v1.1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label: 333, Predict: The <REGISTER_CONTENT> in line sleepy-bijou is <333>., Parsed: 333, prompt length: 4783\n",
      "34it [12:19, 20.81s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 18152, Predict: The <REGISTER_CONTENT> in line wide-eyed-frenzy is <18152>., Parsed: 18152, prompt length: 4827\n",
      "35it [12:43, 21.83s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 49178, Predict: The <REGISTER_CONTENT> in line elite-pottery is <49178>., Parsed: 49178, prompt length: 4789\n",
      "36it [13:04, 21.73s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 37243, Predict: The <REGISTER_CONTENT> in line taboo-cinema is <37243>., Parsed: 37243, prompt length: 4796\n",
      "37it [13:26, 21.66s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 23545, Predict: The <REGISTER_CONTENT> in line scary-kid is <23545>., Parsed: 23545, prompt length: 4782\n",
      "38it [13:47, 21.61s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 7156, Predict: The <REGISTER_CONTENT> in line filthy-eternity is <7156>., Parsed: 7156, prompt length: 4829\n",
      "39it [14:09, 21.61s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 18185, Predict: The <REGISTER_CONTENT> in line lean-machinery is <18185>., Parsed: 18185, prompt length: 4796\n",
      "40it [14:31, 21.84s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 12739, Predict: The <REGISTER_CONTENT> in line fierce-count is <12739>., Parsed: 12739, prompt length: 4794\n",
      "41it [14:52, 21.48s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 30737, Predict: The <REGISTER_CONTENT> in line divergent-front is <30737>., Parsed: 30737, prompt length: 4783\n",
      "42it [15:13, 21.22s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 4193, Predict: The <REGISTER_CONTENT> in line long-perpendicular is <4193>., Parsed: 4193, prompt length: 4836\n",
      "43it [15:33, 21.08s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 47424, Predict: The <REGISTER_CONTENT> in line hilarious-slash is <47424>., Parsed: 47424, prompt length: 4772\n",
      "44it [15:56, 21.45s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 37903, Predict: The <REGISTER_CONTENT> in line auspicious-capacity is <37903>., Parsed: 37903, prompt length: 4819\n",
      "45it [16:18, 21.75s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 27303, Predict: The <REGISTER_CONTENT> in line profuse-pence is <27303>., Parsed: 27303, prompt length: 4789\n",
      "46it [16:40, 21.67s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 49764, Predict: The <REGISTER_CONTENT> in line <tricky-teen> is <49764>., Parsed: 49764, prompt length: 4801\n",
      "47it [17:02, 21.89s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 1086, Predict: The <REGISTER_CONTENT> in line filthy-conviction is <1086>., Parsed: 1086, prompt length: 4840\n",
      "48it [17:23, 21.55s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 4432, Predict: The <REGISTER_CONTENT> in line lush-rice is <4432>., Parsed: 4432, prompt length: 4818\n",
      "49it [17:43, 21.04s/it]Using conversation template: vicuna_v1.1\n",
      "Label: 6729, Predict: The <REGISTER_CONTENT> in line abashed-tart is <6729>., Parsed: 6729, prompt length: 4791\n",
      "50it [18:03, 21.67s/it]\n",
      "************ Finish testing 200 lines per prompt with average prompt length 4805.24, accuracy: 0.96 ************\n"
     ]
    }
   ],
   "source": [
    "!python3 eval.py --model-name-or-path  lmsys/longchat-13b-16k --task lines --longchat_flash_attn --eval_shortest_only"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}