{"id":310,"date":"2023-02-23T01:09:20","date_gmt":"2023-02-23T01:09:20","guid":{"rendered":"https:\/\/freezion.com\/?p=310"},"modified":"2023-02-23T01:34:49","modified_gmt":"2023-02-23T01:34:49","slug":"chatgpt-models-experiment","status":"publish","type":"post","link":"https:\/\/freezion.com\/?p=310","title":{"rendered":"chatGPT Models Experiment"},"content":{"rendered":"\n

chatGPT Models Experiment<\/h3>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n

I\u2019ve been playing with chatGPT lately, and recently started experimenting with it through the OpenAI API<\/a> using python. Since I\u2019m relatively new to AI concepts, I\u2019ve been reading up a bit to try and get up to speed. <\/p>\n\n\n\n

As I did that, I realized that there are 4 different models OpenAI exposes through the API: ada<\/code>, babbage<\/code>, curie<\/code>, and davinci<\/code>. While it was easy enough to understand that the models offer different levels of complexity, it wasn’t at all clear to me what that really means in a practical sense.<\/p>\n\n\n\n

I wanted to understand better the difference between the models and the effect the selection of model has on the output. This can be done using the GPTools Comparison Tool<\/a>, but I wanted something I could control more fully. <\/p>\n\n\n\n

It turns out that doing this is a pretty straight forward process with python: you can simply just run the exact same prompt with the same settings, and change only the model to see how different things are. So, I set up the following code for this task\u200a\u2014\u200ait takes a prompt and runs it through each model with a temperature<\/a> of 0 (to limit the model to the least risk). The response text is output with the name of the model that generated it.<\/p>\n\n\n\n

#!\/bin\/env python3\nimport openai\nimport os\n\nopenai.api_key = os.getenv(\"OPENAI_API_KEY\")\n\n# =====================\n# models\n# =====================\n# text-ada-001\n# text-babbage-001\n# text-curie-001\n# text-davinci-003\n# =====================\nprompt = \"complete this sentence: \\\"This is...\\\"\"\ntemperature = 0\nmax_tokens = 150\ntop_p = 1\nfrequency_penalty = 1\npresence_penalty = 1\n\nada_response = openai.Completion.create(\n  model=\"text-ada-001\",\n  prompt=prompt,\n  temperature=temperature,\n  max_tokens=max_tokens,\n  top_p=top_p,\n  frequency_penalty=frequency_penalty,\n  presence_penalty=presence_penalty\n)\nprint(\"ada:%s\\n\\n\" % ada_response.choices[0].text)\n\nbabbage_response = openai.Completion.create(\n  model=\"text-babbage-001\",\n  prompt=prompt,\n  temperature=temperature,\n  max_tokens=max_tokens,\n  top_p=top_p,\n  frequency_penalty=frequency_penalty,\n  presence_penalty=presence_penalty\n)\nprint(\"babbage:%s\\n\\n\" % babbage_response.choices[0].text)\n\ncurie_response = openai.Completion.create(\n  model=\"text-curie-001\",\n  prompt=prompt,\n  temperature=temperature,\n  max_tokens=max_tokens,\n  top_p=top_p,\n  frequency_penalty=frequency_penalty,\n  presence_penalty=presence_penalty\n)\nprint(\"curie:%s\\n\\n\" % curie_response.choices[0].text)\n\ndavinci_response = openai.Completion.create(\n  model=\"text-davinci-003\",\n  prompt=prompt,\n  temperature=temperature,\n  max_tokens=max_tokens,\n  top_p=top_p,\n  frequency_penalty=frequency_penalty,\n  presence_penalty=presence_penalty\n)\nprint(\"davinci:%s\\n\\n\" % davinci_response.choices[0].text)<\/pre>\n\n\n\n
Running the code above with the sample prompt included in it doesn\u2019t really help much with understanding the differences between the models (although it does sort of highlight how hilariously snarky davinci can appear to be at times):<\/p>\n\n\n\n
ada:
This is a house.

babbage:
This is a book.

curie:
This is a pen.

davinci:
This is a sentence.<\/pre>\n\n\n\n
One reason it\u2019s tough to see a difference is that the task itself is relatively simple, so there\u2019s not much distinction the amount of training the model has had can offer in this instance.<\/p>\n\n\n\n
Giving the AI a more complex task, like brainstorm 5 slogans for an airport.<\/code> for example, and the difference becomes significantly clearer:<\/p>\n\n\n\n
ada:
1. airport is the place to be for a getaway car
2. airport is the place to be when your just looking for some peace and quiet
3. airport is the place to be when you need it the most
4. airport is the place to be for a break from the everyday routine
5. airport is the place to be for a chance against time

babbage:
1. \"Welcome to the airport!\"
2. \"The best airport in the world!\"
3. \"A place where you can be yourself!\"
4. \"The perfect place to fly!\"
5. \"The best airport in the world for travel.\"

curie:
1. Welcome to the Airport of the Future! 
2. We're Making Flying Easy Again! 
3. Fly with Confidence, We've Got You Covered! 
4. The Best Airport in the World is Right Here at Our Airports! 
5. Let's Go Places - With Our Airports by Your Side

davinci:
1. \"Fly with Us and Soar to New Heights!\" 
2. \"Your Gateway to the World\" 
3. \"Where Adventure Takes Off\" 
4. \"The Sky's the Limit at Our Airport\" 
5. \"Take Flight with Us!\"<\/pre>\n\n\n\n
In this case, the creativity between the different models is much clearer.<\/p>\n\n\n\n
A couple of things I noticed going through this exercise: <\/p>\n\n\n\n
    \n
  1. For some reason, the curie model doesn\u2019t wrap it\u2019s output in quotes where the other models do. <\/li>\n\n\n\n
  2. Running the same prompt repeatedly resulted in the exact same responses from the models. It makes sense that this would be the case given a temperature of 0 I suppose, but I\u2019d sort of expected that the results would be different each time I ran the prompt based on my interactions with chatGPT through the website.<\/li>\n<\/ol>\n\n\n\n
    Overall, this helped me understand much better how the different models present very different results in some circumstances, and helped me also understand where maybe they don\u2019t make much of a difference. Since there is a pretty significant cost difference between the models, it\u2019s good to know the latter bit, so I can use the least expensive model to do the types of tasks where the responses don\u2019t differ much between the models.<\/p>\n","protected":false},"excerpt":{"rendered":"
    chatGPT Models Experiment I\u2019ve been playing with chatGPT lately, and recently started experimenting with it through the OpenAI API…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-310","post","type-post","status-publish","format-standard","hentry","category-hacking"],"_links":{"self":[{"href":"https:\/\/freezion.com\/index.php?rest_route=\/wp\/v2\/posts\/310","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/freezion.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/freezion.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/freezion.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/freezion.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=310"}],"version-history":[{"count":5,"href":"https:\/\/freezion.com\/index.php?rest_route=\/wp\/v2\/posts\/310\/revisions"}],"predecessor-version":[{"id":317,"href":"https:\/\/freezion.com\/index.php?rest_route=\/wp\/v2\/posts\/310\/revisions\/317"}],"wp:attachment":[{"href":"https:\/\/freezion.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=310"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/freezion.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=310"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/freezion.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=310"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}