Thanks for the shoutout
@borski !
Depending on how much data there is to parse though, running an LLM wouldn't really be all that expensive.
Vectorizing the entire DB of text into RAG may be a problem though. If we're talking gigabytes of text, the risk of hallucination goes up significantly. And RAG definitely has its limitations.
A much more efficient option in my opinion would be to consider integrating Jina. Jina is capable of scraping and turning any publicly accessible page into pure markdown, which can then be read by even small 8B models with an extremely high degree of accuracy. It would mean zero RAG, no vectorizing, just straight one-shot LLM queries. And since it would be operating in a JIT (just in time) style method, there is no major storage to consider.
Here's a really, really quick PoC.
@joec please don't cringe too hard at my coding skills, that's not my forte!
Bash:
#!/bin/bash
API_KEY="Borski would murder me if he found out I hardcoded an API key" # Your OpenAI API Key
URL="https://r.jina.ai/lucidowners.com/threads/adding-an-llm-support-to-this-forum.10547/#post-233593"
# Get the markdown directly.
markdown=$(curl -s "$URL")
# Escape double quotes and backslashes for JSON
escaped_markdown=$(echo "$markdown" | sed 's/"/\\"/g' | sed 's/\\/\\\\/g') # Escape backslashes as well
# Construct the JSON payload.
json_payload=$(jq -n \
--arg markdown "$escaped_markdown" \
'{model: "gpt-4o", messages: [{role: "system", "content": "You are a helpful assistant."}, {role: "user", "content": "Summarize the following: \($markdown)"}]}')
response=$(curl https://api.openai.com/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $API_KEY" \
-d "$json_payload" --fail)
summary=$(echo "$response" | jq -r '.choices[0].message.content')
echo "Summary:"
echo "$summary"
So what that would give us would be this:
Obviously there's so much more here that can be done, especially with prompting, passing user parameters, etc. There's also the question of dealing with pagination.
Jina uses a client-side API key that gets generated the moment you visit their URL (you don't need an account) so there's nothing to configure on the forum's side.
