forked from ollama/ollama
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signed-off-by: Matt Williams <[email protected]>
- Loading branch information
1 parent
a2405ec
commit 1ade380
Showing
3 changed files
with
110 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,78 @@ | ||
import * as readline from "readline"; | ||
|
||
const model = "llama2"; | ||
type Message = { | ||
role: "assistant" | "user" | "system"; | ||
content: string; | ||
} | ||
const messages: Message[] = [{ | ||
role: "system", | ||
content: "You are a helpful AI agent." | ||
}] | ||
|
||
const rl = readline.createInterface({ | ||
input: process.stdin, | ||
output: process.stdout | ||
}) | ||
|
||
async function chat(messages: Message[]): Promise<Message> { | ||
const body = { | ||
model: model, | ||
messages: messages | ||
} | ||
|
||
const response = await fetch("http://localhost:11434/api/chat", { | ||
method: "POST", | ||
body: JSON.stringify(body) | ||
}) | ||
|
||
const reader = response.body?.getReader() | ||
if (!reader) { | ||
throw new Error("Failed to read response body") | ||
} | ||
const content: string[] = [] | ||
while (true) { | ||
const { done, value } = await reader.read() | ||
if (done) { | ||
break; | ||
} | ||
const rawjson = new TextDecoder().decode(value); | ||
const json = JSON.parse(rawjson) | ||
|
||
if (json.done === false) { | ||
process.stdout.write(json.message.content); | ||
content.push(json.message.content) | ||
// messages.push({role: "system", content: text}) | ||
} | ||
|
||
} | ||
return { role: "assistant", content: content.join("") }; | ||
} | ||
|
||
async function askQuestion(): Promise<void> { | ||
return new Promise<void>((resolve) => { | ||
rl.question("\n\nAsk a question: (press enter alone to quit)\n\n", async (user_input) => { | ||
if (user_input.trim() === "") { | ||
rl.close(); | ||
console.log("Thankyou. Goodbye.\n") | ||
console.log("=======\nHere is the message history that was used in this conversation.\n=======\n") | ||
messages.forEach(message => { | ||
console.log(message) | ||
}) | ||
resolve(); | ||
} else { | ||
console.log(); | ||
messages.push({ role: "user", content: user_input }); | ||
messages.push(await chat(messages)); | ||
await askQuestion(); // Ask the next question | ||
} | ||
}); | ||
}); | ||
} | ||
|
||
async function main() { | ||
await askQuestion(); | ||
|
||
} | ||
|
||
main(); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
{ "dependencies": { "@types/node": "^20.10.4", "prompt-sync": "^4.2.0", "readline": "^1.3.0" } } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
# Simple Chat Example | ||
|
||
The **chat** endpoint is one of two ways to generate text from an LLM with Ollama. At a high level you provide the endpoint an array of objects with a role and content specified. Then with each output and prompt, you add more of those role/content objects, which builds up the history. | ||
|
||
## Review the Code | ||
|
||
You can see in the **chat** function that actually calling the endpoint is done simply with: | ||
|
||
```typescript | ||
const body = { | ||
model: model, | ||
messages: messages | ||
} | ||
|
||
const response = await fetch("http://localhost:11434/api/chat", { | ||
method: "POST", | ||
body: JSON.stringify(body) | ||
}) | ||
``` | ||
|
||
With the **generate** endpoint, you need to provide a `prompt`. But with **chat**, you provide `messages`. And the resulting stream of responses includes a `message` object with a `content` field. | ||
|
||
The final JSON object doesn't provide the full content, so you will need to build the content yourself. In this example, **chat** takes the full array of messages and outputs the resulting message from this call of the chat endpoint. | ||
|
||
In the **askQuestion** function, we collect `user_input` and add it as a message to our messages and that is passed to the chat function. When the LLM is done responding the output is added as another message to the messages array. | ||
|
||
At the end, you will see a printout of all the messages. | ||
|
||
## Next Steps | ||
|
||
In this example, all generations are kept. You might want to experiment with summarizing everything older than 10 conversations to enable longer history with less context being used. |