WIP: Tensorflow Micro based WoV - #11068
Draft
lgirdwood wants to merge 23 commits into
Draft
Conversation
…A capture topology
…and topology usage
…library base address
When CONFIG_LLEXT_TYPE_ELF_RELOCATABLE is active, bypass appending static address flags (-Ttext, --section-start, -Tdata) in the linker helper script. This keeps section base addresses at 0. Also adjust the offset calculator to avoid integer parsing errors when all section addresses are set to 0. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…table modules Implement page-level virtual memory mapping using Zephyr's sys_bitarray utility over the library region. Compile section layout at load-time to allocate virtual addresses and rewrite section sh_addr headers in-place. This enables Zephyr LLEXT to naturally relocate references. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Enable CONFIG_LLEXT_EXPORT_BUILTINS_BY_SLID=y in llext_relocatable.conf to link relocatable LLEXT modules against build-time function signature hashing, providing load-time ABI mismatch protection. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…_mtpm Enable CONFIG_SOF_STAGING=y, CONFIG_CPP=y, CONFIG_STD_CPP17=y, and CONFIG_COMP_TENSORFLOW=m in intel_adsp_ace15_mtpm.conf to allow building the TFLM keyword detection module as an LLEXT module on ARL. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Guard Cadence Tensilica XCC-specific compiler options and tie intrinsic kernels (nn_hifi_lib) under CMAKE_C_COMPILER_ID STREQUAL "Xtensa". When building with open-source Zephyr SDK GCC, use portable TFLM reference kernels and stdc++ to enable building tflm.llext as a relocatable module. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…ation Include <abi.conf> in topology definitions to embed the required IPC4 ABI manifest section (ABI 3.29.1), resolving kernel topology ABI validation. Add sof-arl-cs42l43-l0-cs35l56-l23-tflm.conf to instantiate the real-time TFLM keyword detection pipeline connected to SoundWire DMIC capture on ARL. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Expand SOF_ZEPHYR_VIRTUAL_HEAP_REGION_SIZE to 4MB (0x400000) and increase CONFIG_HEAP_MEM_POOL_SIZE to 32KB (32768) for intel_adsp_ace15_mtpm board. This satisfies dynamic LLEXT module memory requirements for tflm.llext (1.01 MB payload) and prevents -ENOMEM allocation errors in llext_load_elf_data() during section map linking. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…logging Implement stream shutdown summary logging (tflm_log_summary_at_shutdown) in tflm_reset() and tflm_free() to output total inferences, per-category classifications, and KPB trigger events to printk/trace logs at stream end. Add KPB event notification dispatch (tflm_notify_kpb) sending NOTIFIER_ID_KPB_CLIENT_EVT (KPB_EVENT_BEGIN_DRAINING) upon high-confidence keyword detection to initiate pre-roll audio history draining to host PCM stream. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Integrate tflmcly widget into SoundWire feature capture topology and configure multi-pin output routes on drc.1. Add production topology build targets sof-mtl-hda-tflm-kpb and sof-ptl-hda-tflm-kpb for HDA Analog input with Key Phrase Buffer (KPB) pre-roll history management. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…action, and usage Update src/audio/tensorflow/README.md with comprehensive documentation covering: - Dual-path KPB + TFLM Wake-on-Voice architecture - Audio input source abstraction across SoundWire, DMIC, SSP, and HDA - Data Processing (DP) domain vs Low Latency (LL) domain scheduling - Stream shutdown summary event logging ([TFLM STREAM SHUTDOWN SUMMARY]) - Topology v2 build, deployment, and testing usage instructions Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
cavs2.5 has no LLEXT/module-manager support, so build CONFIG_COMP_TENSORFLOW in statically rather than as an LLEXT module (needs C++17/CONFIG_CPP too). Register the DP scheduler at platform init so tflmcly/mfcc can actually run as async Data Processing tasks. Raise ZEPHYR_TWB_STACK_SIZE, heap pool and malloc arena, since TFLM's interpreter/arena allocation and avcodec-style open() calls need well beyond the previous 8KB defaults. Add tflmcly.toml to the PTL rimage manifest gate so it's included whenever tensorflow is built in (static or LLEXT).
…r SDK) nnlib-hifi4's NN kernels are HiFi4 TIE-intrinsic code with no HiFi3 equivalent checked out. Gate on the real core ISA (CONFIG_XTENSA_HIFI4), not just whether the compiler is Clang -- Clang also builds HiFi3 targets (e.g. tgl/cavs2.5), where nn_hifi_lib must not be built or linked, and the HIFI4/NNLIB_V2 kernel defines must not be forced on. Parameterize the previously hardcoded intel_ace15_mtpm toolchain/HAL include paths and -mcpu via SOC_TOOLCHAIN_NAME/ZEPHYR_SDK_INSTALL_DIR so this also works for other SoCs. For a statically-linked (non-LLEXT) image, CONFIG_MINIMAL_LIBC lacks libm and a few libc functions (abs()) that TFLM needs. Linking the toolchain's whole libc.a conflicts with Zephyr's malloc/free and needs an unavailable stderr for __assert_no_args, so instead extract just the specific abs()/ libm archive members TFLM actually calls into a small private tflm_libc_shim archive and link only that; disable assert() via NDEBUG since the toolchain's assert handler isn't usable either. speech.cc/micro_speech_quantized_model_data.cc were already compiled via tflm_lib's own target_sources() list; drop them (and the now-unused math_stubs.c) from the separate add_local_sources(sof ...) call to avoid listing them twice.
…pier mfcc.1 in host-gateway-src-mfcc-tflm-capture.conf had no bytes control at all, so mfcc_prepare() hit comp_get_data_blob() returning cd->config == NULL and failed with "configuration is missing" -- the detection path (mfcc -> tflmcly) has never actually started on hardware before this. Attach a real default blob via HDA_MIC_MFCC_PARAMS (mel40.conf/ mel40_compress.conf, generated by setup_mfcc.m's new 40-bin/20ms-hop profile matching TFLM micro_speech's TFLM_FEATURE_SIZE/STRIDE/DURATION), and thread HDA_MIC_MFCC_PARAMS=default through every sof-hda-tflm build target (mtl/ptl/tgl) in tplg-targets-hda-generic.cmake. Add a real "HDA Mic TFLM Detect" host-copier on pipe 1 (was a virtual out_drv sink with nothing behind it) and route tflmcly's output there instead of the terminal virtual.tflm_sink, plus a matching PCM entry in sof-hda-tflm.conf, so the detection pipeline has a real host-facing device to arm/observe (see README: KPB pin 1 -> detection chain; this pipe's host-copier is for arming/instrumentation, not the pin 2 drain target which remains host-copier.0.capture). Fill in in_channels/out_channels (1, mono) on every audio_format block that was missing it and bump src.1's num_input_audio_formats to include a 16kHz passthrough entry, needed once a real 16kHz-native input format is exercised via the new host-copier. MFCC_FRAME_BYTES corrected 344 -> 184 to match mel40's actual per-hop size: 24-byte mfcc_data_header + 40 x int32 Q9.23 mel values (mel80 was 344; mel40 is what this topology's TFLM_FEATURE_SIZE=40 model actually expects).
tflm_process() was handing tflmcly raw bytes straight from the mfcc-> tflmcly link to the model, reinterpreted as if they were already tightly-packed int8 quantized stock-model features. In reality MFCC's native output is a 24-byte struct mfcc_data_header followed by TFLM_FEATURE_SIZE int32_t Q9.23 mel-log values -- there was no header stripping, no int32->int8 requantization, and no element-count reconciliation anywhere. Once the MFCC config blob is actually attached (previous commit) and mfcc.prepare() stops failing, this is the code that turns its real output into something the classifier can use. Add mfcc_mel_q23_to_int8() to strip the header and requantize each hop's Q9.23 mel values against the model's *real* input tensor scale/ zero_point, and drive the sliding window on MFCC_HOP_BYTES rather than whatever byte count happened to be available. Loop over source_get_data while a full hop is available instead of a single one-shot copy, so frames aren't dropped when more than one hop's worth of data has accumulated between DP task runs. speech.cc/h: expose the model's real input_scale/input_zero_point (previously only output-side scale/zero_point were read) by calling Init_Interpreter() once from TF_InitOps() at prepare time instead of every TF_ProcessClassify() call, and store the raw int8 output (tfc->raw_output[]) alongside the dequantized float predictions so callers can log/compare the pre-dequant values directly. Add OOM checks on the op_resolver/interpreter allocations, which previously had none. Add debug instrumentation ([DBG hop]/[DBG window]/[DBG raw_output]/[DBG quant]) used to root-cause a flat, content-independent model output during on-hardware testing -- real mel values ranged ~0.03-1.5 after Q9.23 normalization vs. the ~0-26 range implied by this model's real input_scale (0.101715)/zero_point(-128), saturating every real input into the bottom ~8% of the int8 range. This is a feature-representation mismatch (this stock model's PCAN-AGC training frontend vs. SOF's linear MFCC mel-log output), not fixed by this commit -- left in place pending a decision on retraining against real SOF MFCC features vs. reimplementing a PCAN-AGC frontend. Also add a static sof_ut_log() shim (was an unresolved extern) and gate llext_entry()/EXPORT_SYMBOL(tflmcly_uuid) behind CONFIG_COMP_TENSORFLOW_MODULE now that this can also build in statically.
…ing docs The previous README described an earlier, partly-aspirational design (virtual.tflm_sink termination, mel-80/344-byte frames, no build-mode distinction) that no longer matches what's actually on this branch after the recent fixes. Update it to reflect reality and add the sections that were missing: - Pipeline diagram and MFCC frame-format description updated for the real wiring: mel-40/184-byte hops, a real host-copier.1 on the detection path (not a terminal virtual sink), and the explicit scheduler_dp_init() call platform.c now makes. - New "Known Limitations / Open Issues" section documenting the PCAN-AGC-vs-linear-MFCC feature-representation mismatch found this session (flat/constant model output on real audio), the 500ms bring-up inference cadence, the unwired IPC4 notification scaffold, and global per-instance state -- so this isn't rediscovered from scratch next time. - New "Build Instructions" section covering both build modes: LLEXT/Clang for HiFi4/5 targets vs. statically-linked/GCC for HiFi3 targets with no LLEXT support (cavs2.5), including the board config and libc-shim details from the static-link build fix. - "Usage" rewritten with a PCM device map (device 0 = real WoV drain target, device 1 = detection arm/observe only) and the hold-open technique for exercising device 1 without triggering arecord's read-error teardown. - New "Training a Custom Keyword Model with Piper-TTS" section: synthetic dataset generation/augmentation, generating training features from SOF's own mel40 MFCC extractor (to sidestep the PCAN-AGC mismatch by construction), training/quantization/conversion, and swapping the generated model into speech.cc/speech.h. - Fixed source-file links, which pointed at another checkout's absolute path (/home/lrg/work/sof-ptl/sof/...); made them relative to this file.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a work-in-progress Wake-on-Voice (WoV) path built around MFCC feature extraction and a TensorFlow Lite Micro (TFLM) classifier, plus the associated topology2 wiring (KPB dual-path drain + detection), build/link support (notably for Xtensa + Clang/LLEXT), and platform memory/stack sizing to make the feature runnable on selected Intel ADSP targets.
Changes:
- Add topology2 definitions for MFCC→TFLM detection pipelines and new topology targets (HDA + SoundWire variants).
- Integrate/extend TFLM classifier implementation and its supporting build system (static vs LLEXT, kernel selection, link helper updates).
- Adjust platform init, memory sizing, and module packaging to support DP-domain inference and larger runtime needs.
Reviewed changes
Copilot reviewed 35 out of 35 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| zephyr/Kconfig | Changes virtual heap sizing defaults. |
| zephyr/CMakeLists.txt | Adds Xtensa+Clang LLEXT linker target/ld-path handling. |
| tools/topology/topology2/sof-hda-tflm.conf | New HDA WoV topology with KPB + MFCC + TFLM routing. |
| tools/topology/topology2/sof-arl-cs42l43-l0-cs35l56-l23-tflm.conf | New ARL SoundWire topology using the MFCC→TFLM capture class. |
| tools/topology/topology2/production/tplg-targets-hda-generic.cmake | Registers new HDA TFLM topology build targets. |
| tools/topology/topology2/production/tplg-targets-ace1.cmake | Registers new ARL TFLM topology build target. |
| tools/topology/topology2/platform/intel/sdw-dmic-audio-feature.conf | Adds TFLM stage into SDW DMIC audio-feature capture pipeline. |
| tools/topology/topology2/include/pipelines/cavs/host-gateway-tdfb-drc-capture.conf | Adjusts DRC widget pin configuration. |
| tools/topology/topology2/include/pipelines/cavs/host-gateway-src-mfcc-tflm-capture.conf | Adds a reusable SRC→MFCC→TFLM capture/detection pipeline class. |
| tools/topology/topology2/include/components/tflm.conf | Defines the topology2 widget class for TFLM classifier. |
| tools/topology/topology2/include/components/mfcc/mel40.conf | Adds mel40 MFCC config blob. |
| tools/topology/topology2/include/components/mfcc/mel40_compress.conf | Adds mel40 MFCC config blob with compress_output enabled. |
| tools/topology/topology2/include/components/kpb.conf | Updates KPB widget UUID in topology2. |
| tools/topology/topology2/include/common/abi.conf | Adds an ABI manifest object definition. |
| tools/rimage/config/tgl.toml.h | Adds conditional inclusion of the TFLM module manifest for TGL images. |
| tools/rimage/config/ptl.toml.h | Adjusts module include gating (currently makes several includes unconditional). |
| src/platform/intel/cavs/platform.c | Initializes DP scheduler when enabled. |
| src/logging/log_backend_probe.c | Adds symbol header include. |
| src/library_manager/llext_manager.c | Adds VMA allocation/layout logic for LLEXT section placement. |
| src/library_manager/llext_manager_dram.c | Frees VMA on restore/unwind paths. |
| src/include/sof/schedule/twb_schedule.h | Increases TWB thread stack size. |
| src/include/sof/lib_manager.h | Stores VMA base/size in module state. |
| src/audio/tensorflow/tflm-classify.c | Major rewrite of TFLM classifier processing, MFCC parsing, KPB triggering, and logging. |
| src/audio/tensorflow/speech.h | Exposes raw output + quantization params to C callers. |
| src/audio/tensorflow/speech.cc | Improves interpreter init lifecycle; exposes input quant params; records raw output. |
| src/audio/tensorflow/README.md | Expands documentation for WoV/TFLM architecture, build, and usage. |
| src/audio/tensorflow/llext/CMakeLists.txt | Refactors kernel selection and C++ toolchain integration for LLEXT builds. |
| src/audio/tensorflow/CMakeLists.txt | Refactors static build, nnlib gating, and toolchain/libc shimming. |
| src/audio/mfcc/tune/setup_mfcc.m | Adds generation of mel40 MFCC config. |
| scripts/llext_offset_calc.py | Handles empty-ELF edge case in size calc. |
| scripts/llext_link_helper.py | Hoists clang target flags; changes section placement/link script generation; strips extra sections. |
| app/llext_relocatable.conf | Enables builtins-by-SLID export setting for relocatable LLEXT. |
| app/boards/intel_adsp_cavs25.conf | Enables C++17 + TensorFlow and increases heap/arena sizing for cavs2.5. |
| app/boards/intel_adsp_ace30_ptl.conf | Enables staging + KPB/TFLM/MFCC/volume/gain modules for PTL. |
| app/boards/intel_adsp_ace15_mtpm.conf | Enables C++17 + TensorFlow module build; increases heap sizing for MTL-PM. |
Suppressed comments (3)
tools/rimage/config/ptl.toml.h:87
- This unconditional include forces the KPB module into PTL images even when CONFIG_COMP_KPB is disabled. It should be gated like other modules in this file.
#include <audio/kpb.toml>
tools/rimage/config/ptl.toml.h:149
- This unconditional include forces the MFCC module into PTL images even when CONFIG_COMP_MFCC is disabled. It should be gated like other modules in this file.
#include <audio/mfcc/mfcc.toml>
tools/rimage/config/ptl.toml.h:175
- This unconditional include forces the TFLM classifier module into PTL images even when CONFIG_COMP_TENSORFLOW is disabled. Other platforms (e.g. tgl.toml.h) guard this with CONFIG_COMP_TENSORFLOW.
#include <audio/tensorflow/tflmcly.toml>
Comment on lines
+225
to
+230
| if not is_relocatable: | ||
| if s_name == '.data': | ||
| command.append(f'-Wl,-Tdata=0x{start_addr:x}') | ||
| else: | ||
| command.append(f'-Wl,--section-start={s_name}=0x{start_addr:x}') | ||
|
|
| #endif | ||
|
|
||
| #if defined(CONFIG_COMP_VOLUME) || defined(LLEXT_FORCE_ALL_MODULAR) | ||
| #include <audio/volume/volume.toml> |
| depends on MM_DRV_INTEL_ADSP_MTL_TLB | ||
| default 0x140000 if VIRTUAL_HEAP_EXTENDED | ||
| default 0x100000 | ||
| default 0x400000 |
Comment on lines
193
to
199
| __cold static int tflm_set_config(struct processing_module *mod, uint32_t param_id, | ||
| enum module_cfg_fragment_position pos, uint32_t data_offset_size, | ||
| const uint8_t *fragment, size_t fragment_size, uint8_t *response, | ||
| size_t response_size) | ||
| { | ||
| struct tflm_comp_data *cd = module_get_private_data(mod); | ||
| struct comp_dev *dev = mod->dev; | ||
| int ret; | ||
|
|
||
| assert_can_be_cold(); | ||
|
|
||
| comp_dbg(dev, "entry"); | ||
|
|
||
| struct sof_ipc4_control_msg_payload *ctl = (struct sof_ipc4_control_msg_payload *)fragment; | ||
|
|
||
| comp_info(dev, "bytes control"); | ||
| ret = comp_data_blob_set(cd->model_handler, pos, data_offset_size, fragment, | ||
| fragment_size); | ||
|
|
||
| /* TODO: now load the model from the blob */ | ||
|
|
||
| return ret; | ||
| return 0; | ||
| } |
Comment on lines
+53
to
+57
| /* MFCC's non-compress output prepends a struct mfcc_data_header (24 bytes) | ||
| * to each hop, followed by TFLM_FEATURE_SIZE int32_t Q9.23 mel-log values | ||
| * (mel40.conf: 40 bins, 20ms hop). This must match MFCC_FRAME_BYTES in | ||
| * host-gateway-src-mfcc-tflm-capture.conf. | ||
| */ |
Comment on lines
+543
to
+546
| LIBS tflm_lib ${TFLM_EXTRA_LIBS} ${TFLM_STD_LIBS} m c gcc | ||
| LIBS_PATH . | ||
| /home/lrg/zephyr-sdk-1.0.1/gnu/xtensa-intel_ace15_mtpm_zephyr-elf/xtensa-intel_ace15_mtpm_zephyr-elf/lib | ||
| /home/lrg/zephyr-sdk-1.0.1/gnu/xtensa-intel_ace15_mtpm_zephyr-elf/lib/gcc/xtensa-intel_ace15_mtpm_zephyr-elf/14.3.0 |
| if(CMAKE_CXX_COMPILER_ID MATCHES "Clang") | ||
| llext_link_options(tflm | ||
| --target=xtensa-intel_ace15_mtpm_zephyr-elf | ||
| --ld-path=/home/lrg/zephyr-sdk-1.0.1/gnu/xtensa-intel_ace15_mtpm_zephyr-elf/bin/xtensa-intel_ace15_mtpm_zephyr-elf-ld |
Comment on lines
+58
to
+61
| "mtl" "platform/intel/mtl.conf" | ||
| "lnl" "platform/intel/lnl.conf" | ||
| "ptl" "platform/intel/ptl.conf" | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP WoV using tensorflow micros speech and MFCC pipeline. Wake word retraining supported via TFLM micro and Piper-TTS
@singalsu to take over.