This is along the lines of what I was considering as a solution.

I was thinking of three solutions:

1. Pass images from camera plugin to another plugin like you do in this PR.
2. Pass the full image byte array from the camera plugin to dart. Then pass it from dart to mlkit plugin.
3. Implement the Camera as part of the flutter framework and design a way to access it on the dart side.

I think the best solution would be three, but we don't have anyone on the team that has time to implement or facilitate such a large change all at once.

My only issue with this PR is that it requires changes in MainActivity, which is what we usually try to avoid. I think to pass images between plugins on the platform side alone would require changing the framework which isn't in my area of expertise.

If you are still in need of an official solution, I just started working on solution 2. I believe it is the simplest to implement of the three and the only drawback is the speed of which the images are scanned would be slow. I should hopefully get that finished and reviewed within the next week or two.

I apologize for keeping you block.

@bparrishMines thanks for the response. I appreciate the value in making sure that this takes the correct shape and so in the interim I've implemented an alternate barcode scanning plugin for my app so that I'm no longer blocked. The alternate plugin just opens up native screens and passes back the scanned code. This is fine, but I'm still motivated to help find a way to get the live barcode scanning into Flutter because I believe that will provide the best experience to developers and end users.

I've been thinking a lot about this over the past few weeks and I believe I'm still partial to solution #1. I think it will deliver the best performance, and the solution I think could be generally useful outside of this use case. I think that there should be a way for one registered native plugin to resolve a reference to another native plugin without modification of the MainActivity in the host application. This would then be a general purpose pathway for all sorts of inter-plugin solutions. This would require an update to the framework to facilitate the resolution of plugin references, but once done, we would have a tool for all plugin developers to utilize and not just something specifically designed for camera access. Method #2 is how I've found that most plugin developers solve this problem. They proxy messages between plugins with their Dart code. In many cases this is just fine. But in some cases it isn't very ergonomic or efficient (as in this case).

I'm motivated and more than willing to help flesh out my idea with modifications to the framework, although I'm not sure where to start with contributions to Flutter proper yet. I'll go see if I can find some contribution docs, and if you have resources you could point me to, I would appreciate it.

@bparrishMines When you have a minute, would you mind looking at this with @dustin-graham? Looks like there are test failures we need to wrap up, and I know Dustin was happy to wrap up work on it. Thanks!

@bparrishMines Just saw your demo at FlutterLive. Great job! How did you decide to handle shuttling data from the Camera to MLKit?

I would still like to be involved in this effort if I can.

@dustin-graham Looks like he does just that, the solution 2 he started working on. Looked pretty fast to me judging from the live show.

You made a real effort with this PR Dustin and I hope you get the chance to chime in on the work of the implementation he got going.

Cheers
Per

@PerLycke Thanks! Yeah the results he showed were very encouraging.

Thanks @dustin-graham !

We implemented it by passing bytes from the camera plugin -> dart -> ML Kit plugin.

The PR's used in the demo were: #900 and #901. But recently, we decided that they could be better implemented and started a second iteration of these PRs.

The process still requires some optimizations, but here is the new PR for the camera side: #965 and the branch for the ML Kit side: master...bparrishMines:mlkit_android_2.

Feedback is always welcomed!

@bparrishMines Looking through the code I notice there's no factory constructor for creating an instance of a FirebaseVisionImage from a CameraImage. Thinking as we're streaming a CameraImage now and not bytes. Any plans for a constructor like that?

@PerLycke I'm leaning towards just keeping the FirebaseVisionImage.fromBytes constructor. I updated the ML Kit code and created a PR: #971. You should now be able to use the bytes constructor from the CameraImage.

I considered a fromCameraImage constructor, but CameraImage is a class in the camera plugin. So, the ML Kit plugin would have to depend on the camera plugin. This would be fine, but our camera plugin requires android version 21+ and I didn't want to limit the ML Kit plugin to use the CameraImage class.

Here is an example of how to use to use it: https://github.com/bparrishMines/mlkit_demo/blob/master/lib/face_expression_reader.dart#L66

It takes quite a bit of code, but I think this does a better job of paving the road for expanding the API.

@dustin-graham Unfortunately, I don't have detailed metrics of passing images. Although, passing a 1920x1080 image from platform -> dart -> platform took about 100ms on a Moto G5 Plus, which is a mid-range android phone from 2017. This suited our needs for the Flutter Live demo.

However, ML Kit recommends using low resolution images anyways, so you could make it faster with smaller images.

I will share a PR soon that would allow people to test out the performance themselves too.

If you need the absolute best performance, forking the camera plugin and adding ML Kit to it would be the best solution. Combining the camera plugin with the ML Kit plugin should also be pretty straight forward.

Also, do you mind if we close this PR and continue talking here: flutter/flutter#19274 or here: #965