re #120

How to avoid unnecessary changes in unmodified files is a secondary question. We will solve that after we agreed on the final version.

But first I need to understand all the changes. I haven't digested your branch yet.

The comments in src/ffi-buffer-clisp.lisp say you got a significant speed up and attribute that to "copying via a single foreign call to MEMORY-AS instead of one foreign call per element via %MEM-REF.' Could you point where this mem-ref per element happens in the old code?

What whould be good to have is self-contained test cases demonstrating the bug in the old code and pass with the new code, and also covering all branches of the copying code changed and introduced. Do you see a sufficienty easy and practical way to implement such tests?

Are you open go have a call to help me understand the pull request?

A call will be great. Sent you email.

This is my test file that reliably demonstrates cl+ssl corrupting memory on clisp.
The bug is fixed by my patch.
Edit the file's *url* to some https one before running it.

; load drakma and base libraries only if not previously loaded
(eval-when (:load-toplevel :compile-toplevel :execute)
  (unless (find-package "DRAKMA")
    (asdf:operate 'asdf:load-op "drakma")))

(use-package "DRAKMA")

; load latest edits of cl+ssl on every load of the file
(asdf:operate 'asdf:load-op "cl+ssl")

(defparameter *url* "https://www.example.com/")

(defun test (nbytes)
  (let ((content
          (make-array nbytes :element-type 'character :initial-element #\.)))
    (http-request *url*
                  :method :post
                  :content #'(lambda (stream) (princ content stream))
                  ;:content content
                  )))

; simpler test. causes exit.
(defun test2 ()
  (let ((cl+ssl:*default-buffer-size* 32))
    (drakma:http-request *url*)))

; these fail but shouldn't
; (test2)
; (test 2049)        ; when content is a lambda
; (test (+ 2048 128)); when content is a string

As for testing the new code, this is a test for the random function,

(defun test-random ()
  (cl+ssl:random-bytes 8))

And this is a test to verify that the various code paths through the new clisp code work correctly.

;;;; tests cl+ssl's ffi-buffer-clisp.lisp's s/b-replace and b/s-replace.
;;;; a successful run is when (test-ffi-buffer-clisp) raises no errors.

(in-package "CL+SSL")
(export '(TEST-FFI-BUFFER-CLISP))

(defun create-test-buffer (length &optional data)
  (let ((result (make-buffer length)))
    (dotimes (i (buffer-length result) result)
      (setf (buffer-elt result i) (if data (aref data i) 0)))))

(defun buffer-equal (expected-vec buf)
  (dotimes (i (length expected-vec) t)
    (unless (equal (aref expected-vec i) (buffer-elt buf i))
      (return nil))))

(defun test-b/s-replace ()
  (mapc #'(lambda (vec expected-buf)
            (mapc #'(lambda (seq)
                      (mapc #'(lambda (*mem-max*)
                                (let ((buf (create-test-buffer 4)))
                                  (unwind-protect
                                    (let ((end (min (buffer-length buf)
                                                    (length seq))))
                                      (b/s-replace buf seq :start1 0 :end1 end
                                                           :start2 0 :end2 end)
                                      (assert (buffer-equal expected-buf buf)))
                                    (release-buffer buf))))
                            (list *mem-max* 2)))
                  (list vec (map 'list #'identity vec))))
        (list #(0 1 2)   #(0 1 2 3) #(0 1 2 3 4))
        (list #(0 1 2 0) #(0 1 2 3) #(0 1 2 3)))
  (values))

(defun test-s/b-replace ()
  (mapc #'(lambda (vec-len buf-data expected-vec)
            (mapc #'(lambda (*mem-max*)
                      (let ((buf (create-test-buffer (length buf-data)
                                                     buf-data))
                            (vec (make-array vec-len
                                             :element-type '(unsigned-byte 8)
                                             :initial-element 0)))
                        (unwind-protect
                          (let ((end (min (buffer-length buf) (length vec))))
                            (s/b-replace vec buf :start1 0 :end1 end
                                                 :start2 0 :end2 end)
                            (assert (equalp expected-vec vec)))
                          (release-buffer buf))))
                  (list *mem-max* 2)))
        (list 2          4          6)
        (list #(0 1 2 3) #(0 1 2 3) #(0 1 2 3))
        (list #(0 1)     #(0 1 2 3) #(0 1 2 3 0 0)))
  (values))

(defun test-ffi-buffer-clisp ()
  (test-b/s-replace)
  (test-s/b-replace))

Sorry these tests aren't more orderly and aren't in line with cl+ssl's testing standard. I did briefly look at whether I could make them so, then decided it was more effort that I was willing to make. So here they are as is, since you asked, for whatever they are worth.

Performance:

I didn't comment that I had achieved a huge speedup. The commenter above me said they had achieved a huge speedup. I was merely hypothesizing that the reason for that speedup couldn't have been lack of copying as he'd claimed because his code also copied. So the reason must have been the style of copying (memory-as instead of %mem-ref). The code before his (i.e. that he improved on) is in cffi's cffi-clisp.lisp (from let's say last week). My present performance improvements over the previous commenter's are not as great as his performance improvements over what came before.

(The reason I say to not look at the very latest cffi code is that I recently submitted a patch so that that too uses memory-as instead of %mem-ref for a large speedup, for the benefit of users who aren't cl+ssl.)

The reason I started working on this was that cl+ssl would crash the process on clisp because ffi-buffer-clisp.lisp had miscalculations about buffer boundaries and was writing into C memory well past array boundaries. I improved performance and fixed the memory leak as side effects of working on that primary bug.

I wonder if the problems addressed in this pull request also caused #163

I'm unfamiliar with the bio code. I've taken a quick look at the write-puts test. I don't think what I've done will directly fix that. However, remembering that stream-write-sequence would on clisp illegally overwrite arbitrary parts of the C memory, if any of the tests called stream-write-sequence then all subsequent behaviour of that process is unpredictable. It'll certainly be worth trying the test suite of 163 to see if its problem has disappeared with this patch. And it's not worth reasoning about any misbehaviour on clisp prior to applying this patch. cl+ssl on clisp is dangerous and should never be used without this patch.

This is how I validate the last commit 45c439f "Stabilise performance for byte-sized operations". *url* needs editing first. Expect output of about one dot, lots of dashes, then a string form of the http response.

; insert the following traces into cl+ssl's streams.lisp to ensure it is called.
; stream-listen
;   (princ #\.) (finish-output)
; stream-read-byte
;   (princ #\-) (finish-output)

(defparameter *url* "https://...")

(defun test-byte ()
  "Exercises stream-listen and stream-read-byte"
  (multiple-value-bind (stream code headers uri stream2 must-close reason)
                       (drakma:http-request *url* :want-stream t)
    (unwind-protect
      (let ((str (chunga:chunked-stream-stream
                   (flexi-streams:flexi-stream-stream stream))))
        (print (list 'str-type (type-of str)))
        (do () ((listen str)))
        (princ
          (with-output-to-string (o)
            (handler-case
                (do ((b (read-byte str nil :eof)
                        (read-byte str nil :eof)))
                    ((eq b :eof))
                  (write-char (code-char b) o)) ;assume character
              (cl+ssl::ssl-error-ssl (e)
                (print e)
                ;SSL_get_error specifies not to call SSL_shutdown
                ;(but now your test leaks C buffer memory.)
                ;really, the error should be trapped by stream-read-byte
                (setf must-close nil))))))
      (when must-close
        (close stream))))
  (values))

Eventual Design Ideal

I see that cffi and cl+ssl are both MIT licenced. Ideally, cffi would deprecate its own wptvd API and copy cl+ssl's improved one from ffi-buffer.lisp and ffi-buffer-clisp.lisp into its own codebase, but would renumber the functions etc. to have a suffix of 2, to differentiate it from its old API. It could then tell its users,

• We are portable across existing Lisps, but make no promises to ever support future Lisps because they may not enable the below
• wptvd2 (therefore) can guarantee to add O(1) overhead
• make-shareable-byte-vector2 (better called make-buffer) returns an object that is to be treated as opaque, i.e. an abstract piece of data. It may only be accessed through its operations in ffi-buffer, i.e. s/b-replace, b/s-replace, buffer-elt, buffer-length. Its elements are of type (unsigned-byte 8). It is not necessarily a (subtype of) Lisp array. Thus, it is not to be operated on with length or replace.
• It may allocate foreign memory so must be released with release-buffer.
• b/s-replace and s/b-replace require a working set of memory that is O(1) as their names suggest. They return their first arguments, just as Lisp's replace does.
• Within the wptvd2 macro body, the C pointer and Lisp variable both refer to the same object in memory and you can read/change its elements via either. (This is not true of the old API with its allowable copy-in/out semantics.)

cl+ssl could then delete its own copy of ffi-buffer.lisp and ffi-buffer-clisp.lisp and use cffi's new API instead. Other users of cffi besides cl+ssl could upgrade to cffi's new API to benefit from the fast (and now correct, faster and O(1) memory) ffi-buffer-clisp.lisp that cl+ssl has long had for itself.

The reason the API is backward incompatible is that (a) make-buffer doesn't necessarily return a subtype of Lisp array and that (b) users who don't call release-buffer may leak memory.

*mem-max* ensures the working set size that s/b-replace and b/s-replace require is O(1) and not proportional to the size of input arrays. It may be defined as any number, but it must be a number. Otherwise that won't be true any more.

The *-replace functions seem inspired by the Lisp replace function, which is assumed to require constant memory, so *-replace should also be made constant memory otherwise their users will be surprised. It can't use constant memory, but it can use constant active (working) memory. Yes, that means the garbage collector must get involved but we can't do better.

If you make the number 2048, future programmers might not understand this and may think it needs to be kept in sync with the other 2048. Also, yes, the buffer code is at a lower level than code that uses it and should be independent of such code (e.g. streams.lisp). It really belongs in cffi. (And may someday get there: cffi/cffi#421 but that's a separate matter.) *-replace should not require memory proportional to their input arrays, which arrays you can imagine someone changing in the future to 1 MiB or 16 MiB, not expecting that every time they call *-replace they are using another 16 MiB of memory (which they may not even have and thus crash), making the replace word very misleading.

Whether the garbage collector collects one piece of data or two is immaterial. What's material is that the two numbers be kept separate, even into the future. Make it 2048 and future programmers are less likely to understand this. But it's fine to make it 2048 if future programmers can be made to understand this some other way. The simplest way I could manage was what I did. What matters is to keep the two numbers separate and not think they must be varied together. Anything that forever keeps *-replace at O(1) working memory is correct.

update: *default-buffer-size* is public so users may make it 128 GiB for some of their streams. *-replace certainly shouldn't require an additional active 128 GiB per call in that case. So *mem-max* is even more important than I'd thought. And it must have a constant value not settable by the user.

My comments about the Eventual Design Ideal:

The cffi:with-pointer-to-vector-data is not a bad API, it's just intended for a different use-case. Namely, for applications which want to call C functions like read or SSL_read directly, and have these functions to directly populate a lisp sequence, with zero intermediate copies. On lisp implementations that provide zero copying cffi:with-pointer-to-vector-data it is an ideal API for that case: give C function a native foreign pointer to the memory underlying a lisp sequence and then work with the sequence in Lisp using standard sequence functions.

In cl+ssl we have a different arrangement: we read into a buffer and then copy from the buffer to another sequence. So we may want better support in CFFI for this use case without deprecating cffi:with-pointer-to-vector-data.

Next, I think there is no need for CFFI to introduce an opaque buffer abstraction.

Applications can just use foreign-alloc and foreign-free for buffer allocation / release, men-ref and mem-aref for element-wise access.

The only thing missing is bulk copy between a foreign memory array and lisp sequence (b/s-relpace, s/b-replace in terms of cl+ssl). If CFFI introduced functions for that, it will be feature complete for the use case we have in cl+ssl.

A big design question is how generic these bulk copy functions should be with respect to lisp sequence type: support only simple vectors, any vectors, or both vectors and lists. This depends on the possibilities we can reasonably expect most lisp implementations can efficiently provide.

Moreover, cffi:with-pointer-to-vector-data, if implemented for a particilar lisp as a zero copy operation, is perfectly suitable for the cl+ssl use case.

The only problem is that for some lisps cffi:with-pointer-to-vector-data performs two redundant copies of the vector data (in the worst case even iterating its elemnts in one by one).

So the question is: can cffi provode more primitive bulk copying operations that are supported on more lisps than currently support the zero copy cffi:with-pointer-to-vector-data.

A relevant doc: https://github.com/cffi/cffi/blob/master/doc/mem-vector.txt

I undersdod what may be the motivation for opaque buffer abstraction in cffi: if efficient bulk co0ying operations are not available in all lisp, then some lisps may implement buffer using shareable-vector/wptvd and other lisps with foreign-alloc / bulk copy.

@vibs29, pleae help me see the O(n^2) run time of s/b-replace prior the commit 61444d (the commit comment says it's one of its fixes). Is it because calling replace repeatedly for sequences of type list and every time increasing :start1 we make replace iterate the list from the start again and again?

PS: I started work on this PR from integrating unit tests based on your examples, but these days I am somewhat busy with other things, so the progress is slow. I will continue in the coming days.

Hi Anton. Yes, exactly right!
Super, thanks for taking a look at all this. A robust cl+ssl for clisp will be great.

I felt a bit guilty about adding commits to the same branch, but didn't know what else to feasibly do. I didn't dare submit my additional tests for the last commit above in the comment. But I think I'll paste it here now just so everything is out there and not sitting privately on my computer where nobody can see it at all. They don't use the official testing framework, which I still haven't taken the time to get the hang of. But they were still useful to me to verify that what I'd finally written did expose (latent) problems in the code before last commit that the last commit solved. It's totally fine to ignore this completely.

(in-package "CL+SSL")

(defun call-with-buffer (size fn)
  (let ((buf (make-buffer size)))
    (unwind-protect
      (funcall fn buf)
      (release-buffer buf))))

(defun expect-error (fn message)
  (assert (not (ignore-errors
                 (funcall fn)
                 t))
          ()
          message))

(defun test-s/b-replace-error-on-large-end1 ()
  (Call-with-buffer
    8
    #'(lambda (buf)
        (expect-error #'(lambda () (s/b-replace (list 0 1 2 3) buf :end1 8))
                      "s/b-replace error expected on bad bounding index"))))

;(replace nil nil) should work but I didn't support it earlier.
(defun test-b/s-replace-none ()
  (mapc #'(lambda (buflen)
            (let ((buf (make-buffer buflen)))
              (unwind-protect
                (mapc #'(lambda (empty)
                          (b/s-replace buf empty
                                       :start1 0 :end1 0 :start2 0 :end2 0))
                      (list '() #()))
                (release-buffer buf))))
        '(4 0)))

;tests b/s-replace when seq end2 is specified to be beyond boundary.
;it should raise an error.
(defun test-b/s-replace-error-on-large-end2 ()
  (call-with-buffer
    2
    #'(lambda (buf)
        (mapc #'(lambda (seq)
                  (expect-error
                    #'(lambda () (b/s-replace buf seq :end2 2))
                    "b/s-replace error expected on bad bounding index"))
              (list '(0) #(0))))))

(defun test-all ()
  (test-s/b-replace-error-on-large-end1)
  (test-b/s-replace-none)
  (test-b/s-replace-error-on-large-end2))