Back

Turning TikTok and Instagram Videos Into Recipes, locally

18 min read · Flutter · Gemma 4 · Local LLMs · Recipe Extraction · Audio Extraction

Cooking content on social media is everywhere, or at least in my feed. My girlfriend and I keep a shared saved list on Instagram, which works great as an endless inspiration board.

But when it actually comes time to cook, scrolling back through a reel to catch that one ingredient you missed, or rewinding because the creator mumbled a measurement, is a pain. I wanted the steps and ingredients in a clear structure I could read at a glance.

That’s why I’m building Mini Chef.

Mini Chef demo: paste a TikTok or Instagram link and get a structured recipe on-device.

I wanted it to work offline, after a one-time model download. No accounts, no subscriptions, no cloud. Gemma 4 turned out to be the right fit. It’s multimodal: it understands text, images, and audio in a single conversation. And because everything runs on the device, the video, the frames, the audio, and the extracted recipe never have to leave the phone.

Why local inference actually matters here

Recipe extraction from video is inherently multi-modal: the model needs to see the frames, hear the audio, and read whatever caption the creator bothered to write. The obvious way to do this is ship everything to a cloud vision API and call it a day. However modern phones are absurdly powerful, they have enough RAM and compute to run a 2-4B parameter model without breaking a sweat. Local LLMs exist. The hardware is there. So let’s use what we already got.

Gemma 4 is the first model family that’s actually small enough to run vision + audio + text reasoning on consumer hardware.

What the app actually does

Paste a TikTok or Instagram link, wait, get a Markdown recipe. Under the hood it’s a pipeline that looks like this:

URL -> normalize & validate -> download with yt-dlp
         │
         ▼
   caption-only attempt (fast path)
         │
         └── fails? -> video preprocessing (frames + audio)
                           │
                           ▼
                     Gemma 4 inference
                           │
                           ▼
                     Markdown parsing
                           │
                           ▼
                     structured Recipe → save locally

All inference happens inside the app through the flutter_gemma package. No network calls after the download.

The main entry point is ImportExtractionService.run(), which either returns a Recipe or a failure message:

// lib/import/import_extraction_service.dart
sealed class ImportExtractionResult {}

final class ImportExtractionSuccess extends ImportExtractionResult {
  ImportExtractionSuccess(
    this.recipe, {
    this.totalMs,
    this.firstTokenMs,
    this.tokenCount,
    this.rawOutputChars,
    this.extractionRuns = const [],
    this.frameTimestamps = const [],
    this.modality,
    this.rawModelOutput,
  });

  final Recipe recipe;
  final int? totalMs;
  final int? firstTokenMs;
  final int? tokenCount;
  final int? rawOutputChars;
  final List<ExtractionRunRecord> extractionRuns;
  final List<double> frameTimestamps;
  final String? modality;
  final RawModelOutput? rawModelOutput;
}

final class ImportExtractionFailure extends ImportExtractionResult {
  ImportExtractionFailure(this.message);
  final String message;
}

Model choice: picking the right Gemma 4 variant

The app supports three variants, chosen automatically based on what the device can actually handle:

Variant Size Modality Typical target
E2B 2.6 GB vision + audio phones
E4B 3.7 GB vision + audio newer phones / tablets
12B 6.5 GB audio only desktop only

The variants live in lib/ai/gemma_model_variant.dart:

// lib/ai/gemma_model_variant.dart
enum GemmaModelVariant {
  e2b(
    id: 'e2b',
    displayName: 'Gemma 4 E2B',
    shortLabel: 'E2B',
    sizeLabel: '2.6 GB',
    litertLmUrl:
        'https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm/resolve/main/gemma-4-E2B-it.litertlm',
    supportsVision: true,
    supportsAudio: true,
  ),
  e4b(
    id: 'e4b',
    displayName: 'Gemma 4 E4B',
    shortLabel: 'E4B',
    sizeLabel: '3.7 GB',
    litertLmUrl:
        'https://huggingface.co/litert-community/gemma-4-E4B-it-litert-lm/resolve/main/gemma-4-E4B-it.litertlm',
    supportsVision: true,
    supportsAudio: true,
  ),
  e12b(
    id: '12b',
    displayName: 'Gemma 4 12B',
    shortLabel: '12B',
    sizeLabel: '6.5 GB',
    litertLmUrl:
        'https://huggingface.co/litert-community/gemma-4-12B-it-litert-lm/resolve/main/gemma-4-12B-it.litertlm',
    supportsVision: false,
    supportsAudio: true,
  );

  static List<GemmaModelVariant> availableOnCurrentPlatform() {
    if (kIsWeb) {
      return const [e2b, e4b];
    }
    if (Platform.isWindows || Platform.isMacOS || Platform.isLinux) {
      return values;
    }
    return const [e2b, e4b];
  }
}

The active variant is configured in GemmaModelConfig:

// lib/ai/gemma_model_config.dart
abstract final class GemmaModelConfig {
  static GemmaModelVariant variant = GemmaModelVariant.e2b;
  static const modelType = ModelType.gemma4;

  static String get downloadUrl =>
      kIsWeb ? variant.webTaskUrl : variant.litertLmUrl;

  static ModelFileType get fileType =>
      kIsWeb ? ModelFileType.task : ModelFileType.litertlm;

  static String get filename => variant.filename;
  static bool get supportsVision => variant.supportsVision;
  static bool get supportsAudio => variant.supportsAudio;
}

One thing that gets easily midded is that you cannot assume every phone can fit a 4B vision model in RAM alongside the rest of your app. So the app probes DeviceProfile.memoryTier and falls back to smaller checkpoints or audio-only when it has to.

DeviceProfile is queried over a platform channel on Android:

// lib/app/device_profile.dart
class DeviceProfile {
  const DeviceProfile({
    required this.totalRamMb,
    required this.availableRamMb,
    required this.gpuRenderer,
  });

  final int totalRamMb;
  final int availableRamMb;
  final String gpuRenderer;

  static Future<DeviceProfile> query() async {
    try {
      final result = await _channel.invokeMethod<Map<dynamic, dynamic>>(
        'getDeviceProfile',
      );
      if (result == null) {
        return const DeviceProfile(
          totalRamMb: 0,
          availableRamMb: 0,
          gpuRenderer: '',
        );
      }
      return DeviceProfile(
        totalRamMb: (result['totalRamMb'] as num?)?.toInt() ?? 0,
        availableRamMb: (result['availableRamMb'] as num?)?.toInt() ?? 0,
        gpuRenderer: (result['gpuRenderer'] as String?) ?? '',
      );
    } catch (e) {
      return const DeviceProfile(
        totalRamMb: 0,
        availableRamMb: 0,
        gpuRenderer: '',
      );
    }
  }

  GemmaMemoryTier get memoryTier {
    if (totalRamMb <= 0) return GemmaMemoryTier.unknown;
    if (totalRamMb <= 6144 || (availableRamMb > 0 && availableRamMb < 1536)) {
      return GemmaMemoryTier.low;
    }
    if (totalRamMb <= 10240 || (availableRamMb > 0 && availableRamMb < 3072)) {
      return GemmaMemoryTier.medium;
    }
    return GemmaMemoryTier.high;
  }
}

Models are downloaded once from Hugging Face (litert-community/gemma-4-*-it-litert-lm) and installed with FlutterGemma.installModel(...).fromNetwork(...).install().

The extraction pipeline

The core orchestration is in lib/import/import_extraction_service.dart. I made it caption-first, since a surprising number of recipe videos have the full recipe in the caption.

1. Caption-only fast path

If the post description is longer than 40 characters and contains recipe-like keywords, the app sends only the caption to Gemma.

// lib/import/import_caption_extraction.dart
sealed class CaptionFirstEligibility {}
final class CaptionFirstEligible extends CaptionFirstEligibility {}
final class CaptionFirstSkipped extends CaptionFirstEligibility {
  CaptionFirstSkipped(this.reason);
  final String reason;
}

abstract final class ImportCaptionExtraction {
  static const minCaptionLength = 40;

  static final _recipeKeywords = [
    'ingredient', 'zutaten', 'direction', 'anleitung',
    'step', 'schritt', 'recipe', 'rezept',
    'tbsp', 'tsp', 'cup', 'g ', 'ml', 'el', 'tl',
  ];

  static CaptionFirstEligibility captionFirstEligibility(String? description) {
    final trimmed = description?.trim();
    if (trimmed == null || trimmed.isEmpty) {
      return CaptionFirstSkipped('empty caption');
    }
    if (trimmed.length <= minCaptionLength) {
      return CaptionFirstSkipped('caption too short (${trimmed.length} chars)');
    }
    if (!_hasRecipeSignal(trimmed)) {
      return CaptionFirstSkipped('no recipe-like signals');
    }
    return CaptionFirstEligible();
  }

  static bool isRecipeDraftAcceptable(Recipe recipe) {
    if (recipe.title.trim().isEmpty) return false;
    if (recipe.ingredients.length < 2 || recipe.steps.isEmpty) return false;

    final namedIngredients = recipe.ingredients
        .where((i) => i.name.trim().isNotEmpty)
        .length;
    if (namedIngredients < 2) return false;

    final nonEmptySteps = recipe.steps
        .where((s) => s.text.trim().isNotEmpty)
        .length;
    return nonEmptySteps >= 1;
  }
}

The result gets accepted only if it parses as a valid recipe and has at least 2 ingredients and 1 step. If the caption is enough, the user gets a recipe without video decoding, no frame extraction, no extra memory pressure.

The orchestrator tries this first and only falls back when it fails:

// lib/import/import_extraction_service.dart
final captionResult = await runCaptionOnly(
  session: session,
  recipeId: recipeId,
  extractor: extractor,
  outputLanguageCode: outputLanguageCode,
  measurementSystem: measurementSystem,
  progress: progress,
  onProgress: onProgress,
  runIndex: runs.length + 1,
);
if (captionResult is ImportExtractionSuccess) {
  runs.addAll(captionResult.extractionRuns);
  return ImportExtractionSuccess(
    _mergeSessionMetadata(captionResult.recipe, session),
    totalMs: captionResult.totalMs,
    firstTokenMs: captionResult.firstTokenMs,
    tokenCount: captionResult.tokenCount,
    rawOutputChars: captionResult.rawOutputChars,
    extractionRuns: runs,
    modality: captionResult.modality,
    rawModelOutput: captionResult.rawModelOutput,
  );
}

2. Video fallback

If the caption doesn’t cut it, the app downloads the media and preprocesses it in lib/ai/video_media_preprocessor_io.dart:

// lib/ai/video_media_preprocessor_io.dart
abstract final class VideoMediaPreprocessor {
  static const defaultMaxFrames = 4;
  static const _frameMaxWidth = 512;

  static Future<VideoPreprocessResult> process(
    String videoPath, {
    void Function(String status)? onProgress,
    List<double>? frameAtSeconds,
    String? audioPath,
    int maxFrames = defaultMaxFrames,
  }) async {
    final duration = await _probeDurationSeconds(videoPath);
    final tempDir = await getTemporaryDirectory();
    final stamp = DateTime.now().millisecondsSinceEpoch;

    onProgress?.call('Extracting audio…');
    final wavPath = '${tempDir.path}/chef_lab_$stamp.wav';
    await _extractWav(videoPath, wavPath);
    final wavBytes = await File(wavPath).readAsBytes();
    await File(wavPath).delete();

    onProgress?.call('Sampling video frames…');
    final timestamps = resolveVideoFrameTimestamps(
      duration: duration,
      markSeconds: frameAtSeconds,
      maxFrames: maxFrames,
    );

    final frameJpegs = <Uint8List>[];
    for (var i = 0; i < timestamps.length; i++) {
      final framePath = '${tempDir.path}/chef_lab_${stamp}_$i.jpg';
      await _extractFrame(videoPath, framePath, timestamps[i]);
      final frameFile = File(framePath);
      if (await frameFile.exists()) {
        frameJpegs.add(await frameFile.readAsBytes());
        await frameFile.delete();
      }
    }

    return VideoPreprocessResult(
      wavBytes: wavBytes,
      frameJpegs: frameJpegs,
      durationSeconds: duration,
      frameTimestamps: timestamps,
    );
  }

  static Future<void> _extractWav(String inputPath, String outputPath) async {
    final quotedIn = _quotePath(inputPath);
    final quotedOut = _quotePath(outputPath);
    final command =
        '-y -i $quotedIn -vn -ac 1 -ar 16000 -c:a pcm_s16le $quotedOut';

    final session = await FFmpegKit.execute(command);
    final returnCode = await session.getReturnCode();
    if (!ReturnCode.isSuccess(returnCode)) {
      throw StateError('ffmpeg audio extraction failed');
    }
  }

  static Future<void> _extractFrame(
    String inputPath,
    String outputPath,
    double atSeconds,
  ) async {
    final ss = atSeconds.toStringAsFixed(2);
    final command =
        '-y -ss $ss -i "$inputPath" -frames:v 1 '
        '-vf scale=$_frameMaxWidth:-2 -q:v 4 "$outputPath"';
    await FFmpegKit.execute(command);
  }
}

Audio becomes a 16 kHz mono WAV (-vn -ac 1 -ar 16000 -c:a pcm_s16le). Frames are up to 3 JPEGs, scaled to a max width of 512 px to keep the vision patch count reasonable.

Frame timestamps default to 15%, 50%, and 85% of the video duration (lib/ai/video_frame_sampling.dart):

// lib/ai/video_frame_sampling.dart
List<double> resolveVideoFrameTimestamps({
  required double duration,
  List<double>? markSeconds,
  int maxFrames = 3,
}) {
  final picked = <double>[];
  final seenCentiseconds = <int>{};

  void add(double seconds) {
    final clamped = duration > 0
        ? seconds.clamp(0.0, duration)
        : seconds < 0 ? 0.0 : seconds;
    final key = (clamped * 100).round();
    if (seenCentiseconds.add(key)) {
      picked.add(clamped);
    }
  }

  final marks = markSeconds;
  if (marks != null && marks.isNotEmpty) {
    for (final seconds in marks) {
      if (picked.length >= maxFrames) break;
      add(seconds);
    }
    if (picked.length == 1 && duration > 0 && picked.length < maxFrames) {
      add(duration * 0.5);
    }
    return picked.take(maxFrames).toList(growable: false);
  }

  const defaultFractions = [0.15, 0.5, 0.85];
  for (final fraction in defaultFractions) {
    if (picked.length >= maxFrames) break;
    final seconds = duration > 0 ? duration * fraction : picked.length.toDouble();
    add(seconds);
  }

  return picked.take(maxFrames).toList(growable: false);
}

That spread usually catches the ingredient spread, the cooking action, and the final plating. Frames get filtered through a quality gate (lib/ai/video_frame_quality.dart) that drops tiny or near-uniform images, because a black frame or a blur helps nobody:

// lib/ai/video_frame_quality.dart
abstract final class VideoFrameQuality {
  static const minJpegBytes = 500;

  static bool areDegenerate(List<Uint8List> frames) {
    if (frames.isEmpty) return true;
    return frames.every(_isDegenerateFrame);
  }

  static bool _isDegenerateFrame(Uint8List bytes) {
    if (bytes.length < minJpegBytes) return true;
    return _isUniform(bytes);
  }

  static bool _isUniform(Uint8List bytes) {
    if (bytes.length < 64) return true;

    final start = bytes.length < 200 ? 20 : 100;
    final end = bytes.length;
    if (end - start < 32) return false;

    var sum = 0;
    var sumSq = 0;
    var count = 0;
    final step = ((end - start) / 64).ceil().clamp(1, 100);
    for (var i = start; i < end; i += step) {
      final value = bytes[i];
      sum += value;
      sumSq += value * value;
      count++;
    }
    if (count < 8) return false;

    final mean = sum / count;
    final variance = sumSq / count - mean * mean;
    return variance < 4 && (mean < 20 || mean > 235);
  }
}

A modality resolver then decides what actually goes to the model:

// lib/import/import_media_modality.dart
enum ImportMediaModality { video, audioOnly }

abstract final class ImportMediaModalityResolver {
  static const minWavBytes = 44;

  static ImportMediaModality resolve({
    required List<Uint8List> frameJpegs,
    GemmaMemoryTier memoryTier = GemmaMemoryTier.unknown,
  }) {
    if (!visionIsAvailable(memoryTier)) {
      return ImportMediaModality.audioOnly;
    }
    if (VideoFrameQuality.areDegenerate(frameJpegs)) {
      return ImportMediaModality.audioOnly;
    }
    return ImportMediaModality.video;
  }

  static bool visionIsAvailable(GemmaMemoryTier memoryTier) {
    if (!GemmaModelConfig.supportsVision) return false;
    return memoryTier != GemmaMemoryTier.low &&
        memoryTier != GemmaMemoryTier.unknown;
  }

  static bool audioIsAvailable(GemmaMemoryTier memoryTier) {
    if (!GemmaModelConfig.supportsAudio) return false;
    return memoryTier != GemmaMemoryTier.low &&
        memoryTier != GemmaMemoryTier.unknown;
  }

  static String audioOnlyExtraContext(String? baseContext) {
    const note =
        'Video frames were unavailable or unsuitable for analysis. '
        'Extract the recipe from the audio track and any caption context above.';
    final base = baseContext?.trim();
    if (base == null || base.isEmpty) return note;
    return '$base\n\n$note';
  }
}

On high- or medium-memory devices, the model sees frames + audio. On low-memory devices or when frames are garbage, it falls back to audio-only. The prompt gets augmented to tell the model it must extract the recipe from the audio track alone.

3. Gemma inference

GemmaRecipeExtractor.collectRecipeOutput() builds a single Message with prompt text, frame bytes, and audio bytes, then streams the response. Generation is kept deterministic enough for structured extraction:

// lib/ai/gemma_recipe_extractor.dart
_chat = await _model!.createChat(
  temperature: 0.4,
  randomSeed: 1,
  topK: 40,
  topP: 0.9,
  tokenBuffer: profile.tokenBuffer,
  supportImage: profile.supportImage,
  supportAudio: profile.supportAudio,
  modelType: GemmaModelConfig.modelType,
  isThinking: false,
  systemInstruction:
      systemInstruction ??
      'You are a recipe extraction assistant. ${RecipeExtractionPrompt.schemaDescription()}',
);

The stream emits TextResponse and ThinkingResponse; the extractor collects both:

// lib/ai/gemma_recipe_extractor.dart
await for (final response in chat.generateChatResponseAsync()) {
  if (tokenCount == 0) {
    firstTokenMs = tokenSw.elapsedMilliseconds;
  }
  tokenCount++;
  if (response is TextResponse) {
    textBuffer.write(response.token);
  } else if (response is ThinkingResponse) {
    thinkingBuffer.write(response.content);
  }
}

The _buildMessage helper assembles the right Message variant depending on whether the run is text, image, audio, or full video:

// lib/ai/gemma_recipe_extractor.dart
Message _buildMessage(GemmaRecipeExtractionInput input, String prompt) {
  switch (input.kind) {
    case GemmaLabMediaKind.text:
      return Message(text: prompt, isUser: true);
    case GemmaLabMediaKind.image:
      return Message.withImage(
        text: prompt,
        imageBytes: input.imageBytes!,
        isUser: true,
      );
    case GemmaLabMediaKind.audio:
      return Message.withAudio(
        text: prompt,
        audioBytes: input.audioWavBytes!,
        isUser: true,
      );
    case GemmaLabMediaKind.video:
      return Message(
        text: prompt,
        images: input.videoFrames!,
        imageBytes: input.videoFrames!.first,
        audioBytes: input.audioWavBytes!,
        isUser: true,
      );
  }
}

4. Parsing into a typed recipe

The prompt forces Gemma into a strict Markdown template:

# <Recipe title>

**Servings:** <number>
**Prep time:** <number> minutes
**Cook time:** <number> minutes

## Tags
## Ingredients
## Steps

RecipeAiParser (lib/ai/recipe_ai_parser.dart) tries the text output, the thinking output, and the combined output, strips stray <|channel> thinking tags, and feeds clean Markdown into RecipeMarkdownParser:

// lib/ai/recipe_ai_parser.dart
abstract final class RecipeAiParser {
  static RecipeParseResult parseGemmaOutput(
    GemmaRecipeOutput output, {
    String? id,
  }) {
    final candidates = output.parseCandidates;
    recipeParseLog.fine('Trying ${candidates.length} parse candidate(s)');
    final failures = <String>[];

    for (var i = 0; i < candidates.length; i++) {
      final cleaned = _stripGemmaChannels(candidates[i]);
      final map = RecipeMarkdownParser.parse(cleaned);
      final result = map == null
          ? RecipeParseFailure('No Markdown recipe found in model output.')
          : _decodeRecipeMap(map, id: id);

      if (result is RecipeParseSuccess) {
        recipeParseLog.info(
          'Parsed recipe id=${result.recipe.id} title="${result.recipe.title}" '
          '(candidate ${i + 1}/${candidates.length})',
        );
        return result;
      }
      failures.add((result as RecipeParseFailure).message);
    }

    return RecipeParseFailure(failures.join(' '));
  }

  static String _stripGemmaChannels(String raw) {
    var text = raw;
    const thoughtStart = '<|channel>thought\n';
    const thoughtEnd = '<channel|>';
    while (true) {
      final start = text.indexOf(thoughtStart);
      if (start < 0) break;
      final end = text.indexOf(thoughtEnd, start);
      if (end < 0) {
        text = text.substring(0, start);
        break;
      }
      text = text.substring(0, start) +
          text.substring(end + thoughtEnd.length);
    }
    return text.trim();
  }
}

RecipeMarkdownParser (lib/ai/recipe_markdown_parser.dart) turns it into a strongly typed Recipe object with ingredients, quantities, units, numbered steps, and optional per-step timers. A small heuristic catches timer hints: “Simmer for 5 minutes” becomes timerSeconds: 300.

// lib/ai/recipe_markdown_parser.dart
static int? _extractTimerSeconds(String stepText) {
  final explicit = RegExp(
    r'\(timer:\s*(\d+)\s*(s|m|min|minutes?)\)',
    caseSensitive: false,
  ).firstMatch(stepText);
  if (explicit != null) {
    final value = _asInt(explicit.group(1)) ?? 0;
    final unit = explicit.group(2)!.toLowerCase();
    return unit == 's' ? value : value * 60;
  }

  final heuristic = RegExp(
    r'(?:for\s+|about\s+|around\s+)?(\d+)\s*(s|sec|seconds?|m|min|minutes?)',
    caseSensitive: false,
  ).firstMatch(stepText);
  if (heuristic != null) {
    final value = _asInt(heuristic.group(1)) ?? 0;
    final unit = heuristic.group(2)!.toLowerCase();
    return unit.startsWith('s') ? value : value * 60;
  }
  return null;
}

The Recipe model is immutable and trivially serializable:

// lib/models/recipe.dart
final class Recipe {
  const Recipe({
    required this.id,
    required this.title,
    required this.servings,
    required this.prepTimeMinutes,
    required this.cookTimeMinutes,
    required this.tags,
    required this.ingredients,
    required this.steps,
    this.sourceUrl,
    this.thumbnailPath,
    this.videoPath,
    this.createdAt,
    this.updatedAt,
  });

  final String id;
  final String title;
  final int servings;
  final int prepTimeMinutes;
  final int cookTimeMinutes;
  final List<String> tags;
  final List<RecipeIngredient> ingredients;
  final List<RecipeStep> steps;
  final String? sourceUrl;
  final String? thumbnailPath;
  final String? videoPath;
  final DateTime? createdAt;
  final DateTime? updatedAt;

  Recipe copyWith({...}) => Recipe(...);
}

Prompt engineering for a local model

Local models are smaller and more sensitive to prompt shape than cloud APIs. I keep the system instruction explicit and short:

// lib/ai/recipe_extraction_prompt.dart
abstract final class RecipeExtractionPrompt {
  static String schemaDescription({
    RecipeMeasurementSystem measurementSystem = RecipeMeasurementSystem.metric,
  }) {
    final unitsRule = RecipeMeasurementPrompt.unitsRule(measurementSystem);
    return '''
Your final reply must be ONLY a Markdown recipe. Do not wrap it in JSON or markdown fences.
Use exactly this structure and no extra commentary outside the recipe:

# <Recipe title>

**Servings:** <number>
**Prep time:** <number> minutes
**Cook time:** <number> minutes

## Tags
- <tag 1>
- <tag 2>

## Ingredients
- <quantity> <unit> <ingredient name>
- <quantity> <unit> <ingredient name> (<optional notes>)

## Steps
1. <step instruction>
2. <step instruction>

Rules:
- $unitsRule
- Each ingredient is one list item starting with "- ".
- Each step is one numbered list item (1., 2., ...).
- Keep the section headers exactly: "## Tags", "## Ingredients", "## Steps".
- If a value is unknown, use servings: 2 and times: 0.''';
  }
}

The user prompt is built in lib/ai/recipe_extraction_prompt.dart and carries:

  • the caption / description,
  • a request to infer ingredients with quantities,
  • a request for numbered steps,
  • a language override (English or German),
  • a measurement-system override (metric or imperial).

Output language and measurement system are first-class in the app, so the same model handles different locales without retraining.

Memory pressure and platform quirks

Running a 2-4B multi-modal model on a phone means the OS is always one memory spike away from killing your app. GemmaMemoryProfile matches the variant to the device tier:

// lib/ai/gemma_memory_profile.dart
factory GemmaMemoryProfile.forDevice(
  DeviceProfile? profile, {
  bool requestedImageSupport = false,
  bool requestedAudioSupport = false,
}) {
  final tier = profile?.memoryTier ?? GemmaMemoryTier.unknown;
  final canUseImage =
      requestedImageSupport && GemmaModelConfig.supportsVision;
  final canUseAudio = requestedAudioSupport && GemmaModelConfig.supportsAudio;

  switch (tier) {
    case GemmaMemoryTier.high:
      return GemmaMemoryProfile._high(
        requestedImageSupport: canUseImage,
        requestedAudioSupport: canUseAudio,
      );
    case GemmaMemoryTier.medium:
      return GemmaMemoryProfile._medium(
        requestedImageSupport: canUseImage,
        requestedAudioSupport: canUseAudio,
      );
    case GemmaMemoryTier.low:
    case GemmaMemoryTier.unknown:
      return GemmaMemoryProfile._low(
        requestedImageSupport: canUseImage,
        requestedAudioSupport: canUseAudio,
      );
  }
}

The low profile is deliberately aggressive: CPU-only, no image, no audio, and a tiny 2048-token context. It is a last-resort safety net rather than a pleasant experience.

An Android trim-memory listener (lib/app/memory_pressure_handler.dart) releases the Gemma model under low/critical memory pressure, unless extraction is currently running:

// lib/app/memory_pressure_handler.dart
class MemoryPressureHandler {
  static const _channel = MethodChannel('com.benjaminraffetseder.minichef/memory');
  static final _listeners = <void Function(int level)>[];

  static void initialize() {
    _channel.setMethodCallHandler((call) async {
      if (call.method == 'onTrimMemory') {
        final level = call.arguments as int? ?? 0;
        for (final listener in List<void Function(int)>.of(_listeners)) {
          listener(level);
        }
      }
    });
  }
}

The extractor defers release if a generation is running, because tearing the model down mid-token would waste the work already done:

// lib/ai/gemma_recipe_extractor.dart
void _onTrimMemory(int level) {
  gemmaExtractLog.fine('onTrimMemory level=$level extracting=$_isExtracting');
  if (level < 10) return;

  if (_isExtracting && level < 15) return;

  if (_isExtracting) {
    _pendingRelease = true;
    return;
  }

  releaseModel().catchError((Object e) {
    gemmaExtractLog.warning('releaseModel on trim memory failed: $e');
  });
}

Data provenance

Every extraction stores full provenance in ImportProvenance (lib/import/import_provenance.dart):

  • source URL and raw yt-dlp metadata,
  • video metadata (title, duration, uploader),
  • chosen model variant,
  • frame timestamps and modality,
  • every extraction attempt (ExtractionRunRecord),
  • raw model output and captured logs.
// lib/import/import_provenance.dart
final class ImportProvenance {
  const ImportProvenance({
    required this.sourceUrl,
    required this.videoId,
    required this.videoTitle,
    this.videoDescription,
    this.videoDurationSeconds,
    this.modelVariant,
    this.totalExtractionMs,
    this.winningExtractionPath,
    this.winningModality,
    required this.mediaPaths,
    this.frameTimestamps = const [],
    this.extractionRuns = const [],
    this.rawMetadata,
    this.rawModelOutput,
    this.capturedLogs = const [],
  });

  final String sourceUrl;
  final String videoId;
  final String videoTitle;
  final String? videoDescription;
  final double? videoDurationSeconds;
  final String? modelVariant;
  final int? totalExtractionMs;
  final String? winningExtractionPath;
  final String? winningModality;
  final MediaPaths mediaPaths;
  final List<double> frameTimestamps;
  final List<ExtractionRunRecord> extractionRuns;
  final Map<String, dynamic>? rawMetadata;
  final RawModelOutput? rawModelOutput;
  final List<CapturedLogEvent> capturedLogs;
}

Useful for debugging the model, but also means you can see exactly how a recipe was derived.

If you want to build something similar

flutter_gemma is the place to start. Install a Gemma 4 model, create a chat, send a multimodal message, stream the response. The API surface is small enough to prototype in an afternoon.

The hard parts, at least in my experience, are everything around the model: getting media onto the device, preprocessing it into the right formats, managing RAM, parsing structured output, and designing a download UX that doesn’t scare people off with a 2.6 GB progress bar.

But if you need on-device AI that sees, hears, and reads, Gemma 4 and flutter_gemma are a genuinely good combination.

A small personal irony

I switched to iOS last week. So now I’m saving up for a MacBook to be able to use my own app again.At least that means a proper macOS version and eventually an iOS build are on the horizon.

Let’s hope Apple doesn’t raise the prices again before I get there.

Thanks for the coffee!

Every cup fuels another line of thoughtful code.

☕×1