Taken from my first attempt at NyxKrage/moondream3-preview-hf, however it needed some changes to work properly with the latest transformers from git, and so the model with the proper config json files can be found at NyxKrage/moondream3-preview-hf-git, though the weights stay the same.

Moondream3 modeling code ported over to follow more proper HuggingFace conventions, code is mostly functional, though there are currently some pretty glaring issues.

1. The whole loop to decode points/regions in the forward pass is obviously not something that should be stay, but I wanted to get the functionality in correctly, so that the flow of how points/bounding boxes are calculated are there as a reference point at least.
2. The code as is currently does not work without caching being enabled (due to simply running the image embeds through the model separately and relying on the kv cache to "store them").
3. Batching with differently sized images does not work, batching with different objectives (query/point/detect) does not work.
4. I was pretty unsure in general on how to properly handle the processor, should it prepend some bogus token to make space for the image embeddings (even though the tokenizer used does not have a dedicated image placeholder token).

Opening this as a draft PR to get some initial feedback on the proper way to handle a most of these as I'm generally unfamiliar with the specifics in terms of how image models should be implemented.
Also the code has the full modeling code rather than using the modular setup for now, because I felt that it would be easier to "fix" the issues, and then identify which parts can be modularized afterwards.

For point 1, from what I couldn't tell if the proper way would be to override the generate method and handle it there, or just have the model handle generation and let the end user worry about the point/bounding box functionality?
For point 2, I think the main pain point, comes from the fact that I ended up handling the image embeddings and shifting around stuff in the modeling code, rather than adding padding for the space the image embeddings should take up in full input embeddings in the processor, but I ended up going for this way just because I was unsure what the correct approach was with the lack of an image token, and so if point 4 gets solved by moving that to the processor, this should come for free I believe.
For point 3, the batching of differently sized images, should just require a bit of tweaking in the reconstruction path for the image projection to handle padding the number of crops, and for the query/point problem, that should hopefully be fixed by the way of solving point 1.