I'd like to share some of my insights from working with OpenAI models on my project. I'm not exactly a tech person, so some of these observations might be obvious to some of you, but I think they're worth sharing for those with less experience or who aren't directly in the field.
Intro:
In early February, my friends and I started a side project where we aimed to build an AI portal called DoMoreAI. For the first two months, we focused on creating an AI tools catalog. Our experiment is based on the idea that in the future, companies will be "Managed by AI, and Driven by Humans." So, our goal was to leave as much as possible to AI and automation
As mentioned before, I'm not a tech guy, but I've been playing with OpenAI models for the past few years, so I had some experience when starting this project.
Tasks We Assigned to AI:
Based on an AI tool's front page, we had the GPT write a one-sentence summary of the AI project + write a more in-depth review of the project, categorize the project into different categories (WHAT category, like blog; TASK category, like writing; FOR category, like content creator), decide if the project offers iOS app, Android app, browser extension, API, find social media links, process information about prices and pricing policy, and more.
Interesting Findings:
- For some projects using AI and automation can 2-10x your output. We added 1000+ AI tools to catalog in 2 months, which won't be possible with a traditional approach.
- When working on a more complex prompt, particularly one with several not directly related tasks, you have to be patient when crafting it. You might eventually find the right wording to achieve the desired results, but it takes time and lots of trial and error. You might even be surprised by what works and what doesn't.
- If cost isn't an issue, you can always break up one complex prompt into several smaller prompts. However, the more requests you send to API, the higher the chance of encountering errors like the 429 error, which may require setting up more sophisticated error handlers for the whole process.
- You need error handlers because, without them, the automation process will suffer.
- With more complex prompts, there are no prompts that always yield the expected results, so you have to plan for what to do if the results aren't satisfactory and how to determine if the result meets your expectations or not.
- GPT-3.0 struggled with outputting JSON strings as requested, but GPT-3.5 is much better at this task. I'd say the number of errors from improperly formatting the response in JSON is 3-4 times lower for GPT-3.5.
- AI models have trouble distinguishing words singular forms from plural forms.
- Just because you can use AI for a given task doesn't mean you always should. Often, standard techniques like using regex can yield better results when extracting something from text than relying solely on AI. A hybrid solution often provides the best results.
- We're using ADA vector embeddings and Pinecone for semantic search in our catalog, and I was really surprised to find that this kind of semantic search works in any language. Even if all the content on our page is in English, you can search in another language and still get decent results.
The Best Mishaps:
- Because of the token limit for requests, we have to ensure that we don't send too long part of the front page to the model. Sometimes, this led to funny situations. If the HTML of the tool's page consists mainly of styles and the model is fed only with styles, then when you ask the AI to write a review of the project, it writes about how beautiful, mobile-friendly, etc., the project is.
- For one project, instead of writing the one-sentence summary, the model's output only included the prompt we were using to generate the summary (needless to say, it was automatically published on our website
)
I hope this post will be useful. We are currently running a campaign on Product Hunt: https://www.producthunt.com/posts/domore-ai
So, if you have any feedback for us or think what we're doing is cool, don't hesitate to support us
[link] [comments]
