-
-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The sliding buffer default size in sse is too short #43
Comments
Easy fix: increase the value
Here is the patch I just developed to fix this: (require '[clojure.core.async.impl.protocols :as impl])
(import '[clojure.lang Counted]
'[java.util LinkedList])
(deftype InfiniteBuffer [^LinkedList buf]
impl/UnblockingBuffer
impl/Buffer
(full? [_this]
false)
(remove! [_this]
(.removeLast buf))
(add!* [this itm]
(.addFirst buf itm)
this)
(close-buf! [_this])
Counted
(count [_this]
(.size buf)))
(defn infinite-buffer []
(InfiniteBuffer. (LinkedList.)))
(defn calc-buffer-size
"- Use stream_buffer_len if provided.
- Otherwise, buffer size should be at least equal to max_tokens
plus the [DONE] terminator if it is provided.
- Else fallbacks on ##Inf and use an infinite-buffer instead"
[{:keys [stream_buffer_len max_tokens]}]
(or stream_buffer_len
(when max_tokens (inc max_tokens))
##Inf))
(defn make-buffer [params]
(let [size (calc-buffer-size params)]
(if (= size ##Inf)
(infinite-buffer)
(a/sliding-buffer size))))
(defn sse-events
"Returns a core.async channel with events as clojure data structures.
Inspiration from https://gist.github.com/oliyh/2b9b9107e7e7e12d4a60e79a19d056ee"
[{:keys [request params]}]
(let [event-stream ^InputStream (:body (http/request (merge request
params
{:as :stream})))
;; ------ Modifications here
events (a/chan (make-buffer params) (map parse-event))]
;; ------ End modifications.
(a/thread
(loop [byte-coll []]
(let [byte-arr (byte-array (max 1 (.available event-stream)))
bytes-read (.read event-stream byte-arr)]
(if (neg? bytes-read)
;; Input stream closed, exiting read-loop
(.close event-stream)
(let [next-byte-coll (concat byte-coll (seq byte-arr))
data (slurp (byte-array next-byte-coll))]
(if-let [es (not-empty (re-seq event-mask data))]
(if (every? true? (map #(a/>!! events %) es))
(recur (drop (apply + (map #(count (.getBytes %)) es))
next-byte-coll))
;; Output stream closed, exiting read-loop
(.close event-stream))
(recur next-byte-coll)))))))
events)) This fixes my buggy test-case above, and when I introduce It took me several days to figure out what was the source of this problem, since I use rather complex core.async code downward the API call and I assumed it was the cause. Note: I posted this issue 5 days ago but my account got sh4d0wbanana5. Took time to figure that out too. |
Thanks for this. Glad you have a workaround for now, I'll find some time to see how best to incorporate your proposed fix. |
Using function calls, maybe because it's a tad slower, I realized I often had problems when parsing streamed results, as some tokens would get dropped along the way.
Explicitly passing
:max_tokens 1000
fixes this issue.openai-clojure/src/wkok/openai_clojure/sse.clj
Lines 29 to 35 in 5c94fb2
I think its too low. My use case is fairly basic. Query, print as characters are appended to a StringBuffer then parse.
Also OpenAI API Reference states the following now, not 16:
The text was updated successfully, but these errors were encountered: