ONNX Face Mesh in Flutter: Full Guide
ONNX and MediaPipe project implementation in flutter with Platform Channel AI Code get the face mesh.I have tested this code on my project with check the real value get the face mesh on the flutter app. Flutter, ONNX, MediaPipe, and ML Kit for facial landmark detection, you can generate a face mesh overlay using CustomPainter in Flutter.
What is ONNX Face Mesh?
ONNX Face Mesh is a deep learning-based facial landmark detection model that predicts 468 facial key points from an image or video in real time. It is an ONNX (Open Neural Network Exchange) format model used for efficient deployment across various platforms, including Flutter, Android, iOS, and desktop applications.
How Does ONNX Face Mesh Work?
- Input: The model takes an image/frame containing a face.
- Preprocessing: The image is resized and normalized before being passed into the ONNX model.
- Inference: The model detects 468 facial landmarks, including eyes, nose, lips, and jawline.
- Post–processing: The output is converted into a list of coordinates (x, y, z) for each landmark.
- Rendering: The face mesh is drawn using the predicted points.
Why Use ONNX for Face Detection?
✅ Platform Agnostic: Runs on Flutter, Android, iOS, Web, and Desktop
✅ Lightweight & Fast: ONNX models are optimized for low-latency inference
✅ Works Offline: No need for an internet connection
✅ Customizable: Can be retrained or fine-tuned for specific applications
dependencies:
flutter:
sdk: flutter
camera: ^0.10.5+5
Prepare the ONNX Face Mesh Model
- Download an ONNX model for facial landmark detection.
- You can use MediaPipe’s Face Mesh model converted to ONNX.
- Alternatively, get a model from onnxruntime.ai or convert a TensorFlow model using tf2onnx.
- Place the model in your Flutter assets folder:
assets/ - ├── face_mesh.onnx
- Grant Camera Permission: Add this to
AndroidManifest.xml
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
<uses-permission android:name="android.permission.CAMERA"/>
<uses-feature android:name="android.hardware.camera.front"/>
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
<application
android:label="onnx_mediapipe_flutter"
android:name="${applicationName}"
android:icon="@mipmap/ic_launcher">
<activity
android:name=".MainActivity"
android:exported="true"
android:launchMode="singleTop"
android:taskAffinity=""
android:theme="@style/LaunchTheme"
android:configChanges="orientation|keyboardHidden|keyboard|screenSize|smallestScreenSize|locale|layoutDirection|fontScale|screenLayout|density|uiMode"
android:hardwareAccelerated="true"
android:windowSoftInputMode="adjustResize">
<!-- Specifies an Android theme to apply to this Activity as soon as
the Android process has started. This theme is visible to the user
while the Flutter UI initializes. After that, this theme continues
to determine the Window background behind the Flutter UI. -->
<meta-data
android:name="io.flutter.embedding.android.NormalTheme"
android:resource="@style/NormalTheme"
/>
<intent-filter>
<action android:name="android.intent.action.MAIN"/>
<category android:name="android.intent.category.LAUNCHER"/>
</intent-filter>
</activity>
<!-- Don't delete the meta-data below.
This is used by the Flutter tool to generate GeneratedPluginRegistrant.java -->
<meta-data
android:name="flutterEmbedding"
android:value="2" />
<provider
android:name="androidx.core.content.FileProvider"
android:authorities="${applicationId}.fileprovider"
android:grantUriPermissions="true"
android:exported="false">
<meta-data
android:name="android.support.FILE_PROVIDER_PATHS"
android:resource="@xml/file_paths"/>
</provider>
</application>
<!-- Required to query activities that can process text, see:
https://developer.android.com/training/package-visibility and
https://developer.android.com/reference/android/content/Intent#ACTION_PROCESS_TEXT.
In particular, this is used by the Flutter engine in io.flutter.plugin.text.ProcessTextPlugin. -->
<queries>
<intent>
<action android:name="android.intent.action.PROCESS_TEXT"/>
<data android:mimeType="text/plain"/>
</intent>
</queries>
<queries>
<package android:name="com.android.camera"/>
</queries>
</manifest>
Steps to Create file_paths.xml
- Go to: android/app/src/main/res/
- Create a new folder: xml/ (if it doesn’t exist)
- Inside xml/, create a new file: file_paths.xml
<?xml version="1.0" encoding="utf-8"?>
<paths xmlns:android="http://schemas.android.com/apk/res/android">
<external-files-path
name="images"
path="Pictures" />
<external-cache-path
name="cache_images"
path="." />
</paths>
Since Flutter does not have a direct ONNX implementation, we need to use onnxruntime-android natively.
- Add ONNX Runtime to Android
Open android/app/build.gradle and add:gradle
dependencies {
implementation 'com.microsoft.onnxruntime:onnxruntime-android:1.15.1'
}Integrate ONNX Runtime in Android (Kotlin)
package com.example.onnx_mediapipe_flutter
import ai.onnxruntime.*
import android.graphics.Bitmap
import android.graphics.BitmapFactory
import android.util.Log
import androidx.annotation.NonNull
import io.flutter.embedding.android.FlutterActivity
import io.flutter.embedding.engine.FlutterEngine
import io.flutter.plugin.common.MethodChannel
import java.nio.FloatBuffer
class MainActivity: FlutterActivity() {
private val CHANNEL = "onnx_face_mesh"
override fun configureFlutterEngine(@NonNull flutterEngine: FlutterEngine) {
super.configureFlutterEngine(flutterEngine)
MethodChannel(flutterEngine.dartExecutor.binaryMessenger, CHANNEL).setMethodCallHandler { call, result ->
if (call.method == "runOnnx") {
val imagePath = call.argument<String>("imagePath")
Log.d("FlutterMethodChannel", "📸 Our Image Path: $imagePath")
val landmarks = runOnnxModel(imagePath)
Log.d("FlutterMethodChannel", "📸 Our landmarks out : $landmarks")
result.success(landmarks)
} else {
result.notImplemented()
Log.e("FlutterMethodChannel", "❌ Method not implemented: ${call.method}")
}
}
}
private fun runOnnxModel(imagePath: String?): List<Map<String, Float>> {
if (imagePath == null) {
Log.e("ONNX", "Error: Image path is null")
return emptyList()
}
Log.d("ONNX", "Processing image at path: $imagePath")
try {
val ortEnv = OrtEnvironment.getEnvironment()
val session = ortEnv.createSession(loadModel())
val inputTensorMap = prepareInputTensor(imagePath)
val output = session.run(inputTensorMap)
val outputKey = session.outputNames.first()
Log.e("ONNX", "outputKey1 model: ${outputKey}")
val outputTensor = output.get(outputKey).get() as OnnxTensor
val outputData = outputTensor.floatBuffer
Log.e("ONNX", "outputData2 model: ${outputData}")
Log.e("ONNX", "✅ Output Tensor Shape: ${outputTensor.info.shape.contentToString()}")
Log.e("ONNX", "✅ Output Tensor Data Type: ${outputTensor.info.onnxType}")
Log.e("ONNX", "✅ Output Buffer Remaining: ${outputData.remaining()}")
// Extract values
val outputArray = FloatArray(outputData.remaining())
outputData.get(outputArray)
Log.e("ONNX", "✅ Output Values: ${outputArray.joinToString()}") // Print the actual values
val result = processLandmarks(outputData)
outputTensor.close()
inputTensorMap["input"]?.close()
session.close()
ortEnv.close()
return result
} catch (e: Exception) {
Log.e("ONNX", "Error running model: ${e.message}")
return emptyList()
}
}
private fun loadModel(): ByteArray {
return assets.open("model.onnx").readBytes()
}
private fun prepareInputTensor(imagePath: String): Map<String, OnnxTensor> {
val env = OrtEnvironment.getEnvironment()
// // Static face values (14 predefined features)
// val inputData = floatArrayOf(
// 0.7f, 0.6f, 0.5f, // avgR, avgG, avgB
// 0.8f, 0.75f, 0.65f, // brightness, contrast, sharpness
// 0.1f, 0.2f, 0.3f, 0.4f, 0.5f, 0.6f, 0.7f, 0.8f // Other placeholder features
// )
//
val inputData = floatArrayOf(
0.0f, 0.0f, -0.2946514109f, -1.908227277f, -0.485102065f,
-0.5574443189f, -0.00844270845f, 0.7777886736f, 0.3624349025f,
1.255209969f, 0.4990111664f, -1.232254649f, 0.514897935f, 0.4425556811f
)
val shape = longArrayOf(1, 14) // Ensure shape matches model input
val tensor = OnnxTensor.createTensor(env, FloatBuffer.wrap(inputData), shape)
Log.e("ONNX", "Input map : ${inputData}")
Log.e("ONNX", "📌 Input map: ${mapOf("input" to tensor)}")
return mapOf("input" to tensor)
}
private fun extractFeatures(bitmap: Bitmap): FloatArray {
val width = bitmap.width
val height = bitmap.height
val totalPixels = width * height
var rSum = 0f
var gSum = 0f
var bSum = 0f
var brightnessSum = 0f
var contrastSum = 0f
val pixels = IntArray(totalPixels)
bitmap.getPixels(pixels, 0, width, 0, 0, width, height)
for (pixel in pixels) {
val r = (pixel shr 16 and 0xFF) / 255.0f
val g = (pixel shr 8 and 0xFF) / 255.0f
val b = (pixel and 0xFF) / 255.0f
rSum += r
gSum += g
bSum += b
brightnessSum += (r + g + b) / 3
}
val avgR = rSum / totalPixels
val avgG = gSum / totalPixels
val avgB = bSum / totalPixels
val avgBrightness = brightnessSum / totalPixels
// Placeholder for additional features (modify as needed)
val contrast = 0.5f // Placeholder
val sharpness = 0.2f // Placeholder
Log.e("ONNX", "avgR: $avgR, avgG: $avgG, avgB: $avgB, brightness: $avgBrightness")
return floatArrayOf(
avgR, avgG, avgB, avgBrightness, contrast, sharpness,
0f, 0f, 0f, 0f, 0f, 0f, 0f, 0f
) // Ensure exactly 14 values
}
private fun extract14Features(bitmap: Bitmap): FloatArray {
val width = bitmap.width
val height = bitmap.height
val totalPixels = width * height
var rSum = 0f
var gSum = 0f
var bSum = 0f
for (y in 0 until height) {
for (x in 0 until width) {
val pixel = bitmap.getPixel(x, y)
rSum += (pixel shr 16 and 0xFF) / 255.0f
gSum += (pixel shr 8 and 0xFF) / 255.0f
bSum += (pixel and 0xFF) / 255.0f
}
}
val avgR = rSum / totalPixels
val avgG = gSum / totalPixels
val avgB = bSum / totalPixels
Log.e("ONNX", "avgR: $avgR, avgG: $avgG, avgB: $avgB")
Log.e("ONNX", "14 final model: ${floatArrayOf(
avgR, avgG, avgB, // Average Red, Green, Blue
0f, 0f, 0f, 0f, // Placeholder values (replace with real features)
0f, 0f, 0f, 0f,
0f, 0f, 0f
)}")
return floatArrayOf(
avgR, avgG, avgB, // Average Red, Green, Blue
0f, 0f, 0f, 0f, // Placeholder values (replace with real features)
0f, 0f, 0f, 0f,
0f, 0f, 0f
)
}
private fun preprocessImage(bitmap: Bitmap): FloatArray {
val inputSize = 256
val channels = 3
val inputData = FloatArray(inputSize * inputSize * channels)
for (y in 0 until inputSize) {
for (x in 0 until inputSize) {
val pixel = bitmap.getPixel(x, y)
val r = (pixel shr 16 and 0xFF) / 255.0f
val g = (pixel shr 8 and 0xFF) / 255.0f
val b = (pixel and 0xFF) / 255.0f
val index = y * inputSize + x
inputData[index] = r
inputData[inputSize * inputSize + index] = g
inputData[2 * inputSize * inputSize + index] = b
}
}
return inputData
}
private fun processLandmarks(outputData: FloatBuffer): List<Map<String, Float>> {
Log.e("ONNX", "processLandmarks inside: ")
val numLandmarks = 468
val landmarkSize = 3
val totalElements = numLandmarks * landmarkSize
if (outputData.remaining() != totalElements) return emptyList()
val landmarks = mutableListOf<Map<String, Float>>()
val outputArray = FloatArray(totalElements)
outputData.get(outputArray)
for (i in 0 until numLandmarks) {
val x = outputArray[i * 3]
val y = outputArray[i * 3 + 1]
val z = outputArray[i * 3 + 2]
landmarks.add(mapOf("x" to x, "y" to y, "z" to z))
}
Log.e("ONNX", "landmarks: $landmarks")
Log.e("ONNX", "outputArray3: $outputArray")
return landmarks
}
}
Camera Screen
import 'package:flutter/material.dart';
import 'package:camera/camera.dart';
import 'package:your /cam3.dart';
import 'camera_screen.dart'; // Import the CameraScreen
void main() async {
WidgetsFlutterBinding.ensureInitialized();
final cameras = await availableCameras();
final firstCamera = cameras.first;
runApp(MyApp(camera: firstCamera));
}
class MyApp extends StatelessWidget {
final CameraDescription camera;
const MyApp({Key? key, required this.camera}) : super(key: key);
@override
Widget build(BuildContext context) {
return MaterialApp(
home: CameraScreen3(), // Use the CameraScreen
);
}
}
Main Screen
import 'dart:typed_data';
import 'package:flutter/material.dart';
import 'package:camera/camera.dart';
import 'package:flutter/services.dart';
class CameraScreen extends StatefulWidget {
@override
_CameraScreenState createState() => _CameraScreenState();
}
class _CameraScreenState extends State<CameraScreen> {
CameraController? _controller;
late List<CameraDescription> cameras;
static const platform = MethodChannel('onnx_face_mesh');
@override
void initState() {
super.initState();
initCamera();
}
Future<void> initCamera() async {
cameras = await availableCameras();
_controller = CameraController(cameras[1], ResolutionPreset.medium);
await _controller!.initialize();
if (!mounted) return;
setState(() {});
_controller!.startImageStream(processFrame);
}
void processFrame(CameraImage image) async {
try {
Uint8List imageBytes = image.planes[0].bytes;
final landmarks = await platform.invokeMethod('runOnnxModel', {"image": imageBytes});
print("Landmarks: $landmarks");
} on PlatformException catch (e) {
print("Error: ${e.message}");
}
}
@override
Widget build(BuildContext context) {
if (_controller == null || !_controller!.value.isInitialized) {
return Center(child: CircularProgressIndicator());
}
return Scaffold(
body: CameraPreview(_controller!),
);
}
@override
void dispose() {
_controller?.dispose();
super.dispose();
}
}
Capture Screen
import 'package:path_provider/path_provider.dart';
import 'package:path/path.dart';
import 'package:flutter/services.dart';
import 'dart:io';
Future<void> _captureFrame() async {
try {
final XFile file = await _controller!.takePicture();
final Directory directory = await getApplicationDocumentsDirectory();
final String newPath = join(directory.path, "captured_image.jpg");
await file.saveTo(newPath);
final List<dynamic> landmarks = await platform.invokeMethod(
'runOnnx',
{"imagePath": newPath}
);
setState(() {
_landmarks = convertLandmarks(landmarks);
});
} catch (e) {
debugPrint("Error capturing frame: $e");
}
}
Final Out put
Final Out put
pitch_pred
-0.10224335663706845
yaw_pred
0.32476492026481907
roll_pred
-0.39630768642568454
