板卡介绍

本次活动使用到的板卡是Syntiant的TinyML Board，它配合EDGE IMPULSE可以轻松建立机器学习模型，而无需任何专用硬件。

板子搭载的主要硬件如下：

· 超低功耗 Syntiant ® NDP101神经决策处理器™

· 主控为Atmel的SAMD21，Cortex-M0+ 32位低功耗48MHz ARM MCU，内置256KB FLASH和32KB SRAM

· 2MB串行闪存

· 一个用户定义的RGB LED

· 一个uSD卡槽

· BMI160 6轴运动传感器

· SPH0641LM4H 麦克风

除此之外，TinyML Board兼容Arduino MKR系列板卡，有5个数字IO，包括一路UART和一路I2C接口。

 

开发环境

arduino-cli 18.0版本

Arduino IDE和EDGE IMPULSE

 

任务介绍

本次活动我选择了任务一：

使用板载麦克风收集音频数据，然后上传到EDGE IMPULSE建立机器学习模型并训练，实现中文语音指令控制板载LED灯。一共实现了三个指令控制，分别是：“打开灯”、“关闭灯”、“灯闪烁”。

因为本次活动使用了Arduino来开发，提供了许多库可供使用，再加上EDGE IMPULSE提供了一份示例工程，而且MCU端的任务是控制LED，相对简单，所以其代码相对简单。而本次任务的重点难点则是在EDGE上建立识别效果良好的机器学习模型上。

 

任务实现

1.数据收集

数据集分为训练集和测试集，EDGE IMPULSE推荐分别占80%和20%。因为实时分类无法使用，所以测试集还是有必要建立，可以初步检测模型效果，避免了模型部署到板子上才发现效果不佳而浪费时间。

在我的模型中，我设置了四个类，除三个语音指令之外还有一个否定类标签（z_openset），用于存放所有噪音数据。四个类分别是：openlight（对应指令“打开灯”）、closelight（对应指令“关闭灯”）、lightflash（对应指令“灯闪烁”）、z_openset。

每个类都有一千条以上的数据，所有数据一共有4小时6分10秒。

三个语音指令的数据主要来源于我自己，剩下的来自于身边的同学朋友。openlight比另外两个指令多是因为，这是我第一个录的指令，没有仔细规划数据量。

z_openset类的数据主要来自于官方的噪音集和我自己录制各种噪音。

官方的噪音集是来自于“Go”和“Stop”的示例项目。在该项目的教程指引中提到，它基于Google Speech Commands数据集的子集而构建，同时还添加了来自Microsoft Scalable Noisy Speech数据集的噪声。包含四个类，每个类有25分钟的数据：“Yes”、“No”、“Unknown”（其他单词）、“Noise”（背景或静态噪声）。

而我自己录制的噪声主要包括生活中的各种噪声（犬吠声、歌曲声、孩童玩闹声、乐器声等等）和中文语音噪声（一段语音广播音频和一段小说讲述音频）。

数据分布图：

2.模型建立

时间序列（Time series data）板块中，窗口大小是968ms，这是因为NDP101输入的每个张量有1600个特征，纵轴有40个频率，横轴有40个FFT窗口。每个FFT窗口采样512次，而麦克风的采样率是16KHz，每个FFT窗口时间就是32ms。每次采样32ms，第一次不重复，而后面的每一次采样，都和前一次重复8ms，所以每个张量的时间窗口就是32+39*24=968ms。因为张量的特征数必须为1600，所以968ms是固定的，无法调整。

而窗口移动（Window increase）指窗口往后移动的距离。这个值小一点更好，值越小，相同时间内执行的推理次数越多，相应的，理论上识别率应该会更高，我将这里设置为了100ms。

3.模型训练

Impulse创建好后，在Syntiant信号处理块中生成特征，此处的参数不需要调整。然后在NN Classifier中开始训练，训练前的参数调整参考官方文档。生成特征和模型训练都需要一定的时间，数据越多花费时间越长。

4.模型测试

 进行模型参数，初步检测模型效果，如果分类效果太差，就要进行检查，查看数据，然后重新训练。而通过模型检测后，就可以部署模型到开发板上了。

 

代码实现

1. 初始化

整个程序的初始化由syntiant_setup()完成。

void setup(void)
{
    syntiant_setup();
}

// Arduino System Setup routine
// Initialises devices. Reads flash device to see if valid uilib has
// been programmed.
// Loads uilib if present
void syntiant_setup(void)
{
    timer4.enable(false); //disable timer4

    // uilib variables
    int s;
    uint32_t v;
    byte i;
    char name[9];
    char *namep = name;

    // Initialize Serial Port
    Serial.begin(115200);
    Serial2.begin(115200);
    //delay(3000); // Enable serial ports to print to console

    // Show sign of life
    pinMode(LED_BUILTIN, OUTPUT); // RED LED

    digitalWrite(LED_BUILTIN, HIGH); // Light RED LED
    //digitalWrite(LED_BLUE, HIGH); // Light BLUE LED
    //digitalWrite(LED_GREEN, HIGH); // Light GREEN LED

    SerialFlash.begin(FLASH_CS);
    SerialFlash.readID(FlashType);
    if(FlashType[0] == SST25VF016B) { // Set up clock for Bluebank board
        analogWrite(3, 0x10); //TinyML Final Board
        REG_TCC1_PER = 1464;
        while (TCC1->SYNCBUSY.bit.PER)
            ;
        REG_TCC1_CC1 = 732;
        while (TCC1->SYNCBUSY.bit.CC1)
            ;
        PORSTB = 24;
    } else if(FlashType[0] == MX25R6435FSN) { // Set up clock for Tessolve board
        // Set up 32KHz NDP clock
        /********************* Timer #3, 16 bit, toggles pin PA18 */
        zt3.configure(TC_CLOCK_PRESCALER_DIV1,      // prescaler
                      TC_COUNTER_SIZE_16BIT,        // bit width of timer/counter
                      TC_WAVE_GENERATION_MATCH_FREQ // frequency or PWM mode
        );
        zt3.PWMout(true, 0, 10);                       // Actually toggles pin PA18
        zt3.setCompare(0, (48000000 / 32000 / 2) - 1); // 32KHz output
        zt3.enable(true);
        PORSTB = 3;
    } else {
        Serial2.print("Unknown board: ");
        Serial2.println(FlashType[0], HEX);
        Serial.print("Board not recognized. Press your board's reset button to exit");
        while(true)
            ;
    }

    // reset NDP
    pinMode(PORSTB, OUTPUT);
    digitalWrite(PORSTB, LOW);
    delay(100);
    digitalWrite(PORSTB, HIGH);

    // Set up SPI (NDP) & SPI1 (SD card)
    SPI.begin();
    SPI.beginTransaction(SPISettings(spiSpeedGeneral, MSBFIRST, SPI_MODE0));

    // See which board we are by trying to read NDP101 Registion Register
    pinMode(NDP9101_CS, OUTPUT);
    digitalWrite(NDP9101_CS, HIGH);
    pinMode(TINYML_CS, OUTPUT);
    digitalWrite(TINYML_CS, HIGH);

    SPI_CS = NDP9101_CS;
    NDP.spiTransfer(NULL, 0, 0x0, NULL, spiData, 1);

    if (spiData[0] == 0x20)
    {
        SPI_CS = NDP9101_CS;
        idle = NDP9101_USB_IDLE;
    }

    else
    {
        SPI_CS = TINYML_CS;
        NDP.spiTransfer(NULL, 0, 0x0, NULL, spiData, 1);

        if (spiData[0] == 0x20)
        {
            SPI_CS = TINYML_CS;
            pinMode(NDP9101_CS, INPUT); // make NDP9101_CS input to save power

            Serial2.begin(115200);

            // Serial2 will be available on TinyML connector pin 6 (RX) & pin 7 (TX)
            // PA20 Arduino pin 6 is RX.
            // PA21 Arduino pin 7 is TX.
            // Assign pins PA20 & PA21 to SERCOM functionality.
            pinPeripheral(6, PIO_SERCOM_ALT);
            pinPeripheral(7, PIO_SERCOM_ALT);
            delay(100);
            Serial2.println("Hello Serial2 World!");

            pinMode(LED_BLUE, OUTPUT);
            pinMode(LED_GREEN, OUTPUT);
            pinMode(USER_SWITCH, INPUT_PULLUP);

            // find which Serial Flash device is connected
            SerialFlash.begin(FLASH_CS);
            SerialFlash.readID(FlashType);
            // Set up pin to drive 5v out to Arduino companion
            if (FlashType[0] == SST25VF016B)
            {
                digitalWrite(ENABLE_5V, LOW); // disable 5v output Bluebank Board
                Serial2.println("Syntiant TinyML Board B");
            }
            if (FlashType[0] == MX25R6435FSN)
            {
                digitalWrite(ENABLE_5V, HIGH); // disable 5v output Tesolve Boards
                Serial2.println("Syntiant TinyML Board T");
            }

            pinMode(PMU_OTG, OUTPUT);   // set up OTG/PMU current pin
            pinMode(ENABLE_5V, OUTPUT); // Set up 5v gate

            // Initialise SGM41512
            if (!PMIC.begin()) {
                Serial2.println("Failed to initialize PMIC!");
            }
            pmuCharge(); // Enable PMU Charge mode

            pinMode(0, OUTPUT);
            pinMode(1, OUTPUT);
            idle = TINYML_USB_IDLE;
        }

        else
        {
            Serial.println("No NDP device found");
            while (true)
                ;
        }
    }

    // Initialize SD & Serial Flash. Try & load NDP BIN file which contains NDP firmware & Neural Network
    // If not able to load bin file, use Bridging Mode to access NDP
    NDP.setInterrupt(NDP_INT, ndpInt);
    switch (loadModel(model))
    {
    case BIN_LOAD_OK:
        ei_printf("BIN file loaded correctly from SD Card");
        runningFromFlash = 1;
        digitalWrite(LED_BLUE, HIGH); // Light BLUE LED as uilib load successful
        loadedFromSD = 1;
        loadedFromSerialFlash = 0;
        break;
    case NO_SD:
        ei_printf("No SD Card inserted, please insert card");
        ei_printf("Running in Bridge Mode");
        break;
    case SD_NOT_INITIALIZED:
        ei_printf("SD Card initialization failed!");
        ei_printf("Running in Bridge Mode");
        break;
    case BIN_NOT_OPENED:
        // ei_printf(model + " NOT opened. Make sure you're using the correct BIN file name.");
        ei_printf("Running in Bridge Mode");
        break;
    case ERROR_LOADING_FLASH:
        ei_printf("Error loading bin from Flash!");
        ei_printf("Running in Bridge Mode");
        break;
    case ERROR_LOADING_SD:
        ei_printf("Error loading bin from SD!");
        ei_printf("Running in Bridge Mode");
        break;
    case LOADED_FROM_SERIAL_FLASH:
        ei_printf("BIN File Loaded correctly from Serial Flash");
        runningFromFlash = 1;
        //digitalWrite(LED_GREEN, HIGH); // Light GREEN LED as uilib load successful
        loadedFromSerialFlash = 1;
        loadedFromSD = 0;
        break;
    default:
        ei_printf("Running in Bridge Mode");
        break;
    }

    // Allow some peripherals to be active in Standby mode.
    // Standby is used when battery powered for lowest power
    TC3->COUNT16.CTRLA.bit.RUNSTDBY = 1; // enable timer3 in sleep mode. This generates the NDP clock
    SYSCTRL->XOSC32K.bit.RUNSTDBY = 1;   // Run the 32KHz Oscillator in sleep
    SYSCTRL->DFLLCTRL.bit.RUNSTDBY = 1;  // Run the Oscillator DFLL in sleep
    SYSCTRL->VREG.bit.RUNSTDBY = 1;      // Run the voltage regulator in sleep. This keeps NDP CLK at 32KHz

    // turn LED off
    digitalWrite(LED_BUILTIN, LOW);

    if (!runningFromFlash) {
        // Reset the NDP if the log load failed
        pinMode(PORSTB, OUTPUT);
        digitalWrite(PORSTB, LOW);
        delay(100);
        digitalWrite(PORSTB, HIGH);

        // Light RED LED as uilib NOT loaded successfully
        digitalWrite(LED_RED, HIGH);
    }

    // Set up timer to turn LEDs off after 1 second
    // ledTimerCount = 1000; // set LED timer for 1 second
    ledTimerCount = 1 * (1000000 / timer_in_uS); // set LED timer for 1 second

    // Set interrupt priority.
    // The priority specifies the interrupt priority value, whereby
    // lower values indicate a higher priority.
    // The default priority is 0 for every interrupt. This is the highest
    // possible priority.
    NVIC_SetPriority(TC4_IRQn, 3); // Make timer 4 the lowest priority

    tankSize = indirectRead(DSP_CONFIG_TANK) >> 4;
    tankAddress = indirectRead(DSP_CONFIG_TANKADDR);

#if defined(WITH_AUDIO)
    // Load Audio Buffer with test pattern
    for (i = 0; i < sizeof(audioBuf) / 2; i++) {
        audioBuf[i] = 4000 * (i - (sizeof(audioBuf) / 4));
    }
#endif


#if defined(WITH_AUDIO)
    AudioUSB.getShortName(namep); // needed for platformio to load AudioUSB
#endif


    //ENABLE_PDM = 25 is  declared in NDP_GPIO.h is the pin for controlling buffer SGM7SZ125. This buffer
    // will be activated (low) for voice command spotting and deactvated (high) for sensor appliactions

    pinMode(ENABLE_PDM, OUTPUT);
#if defined(WITH_IMU)
    digitalWrite(ENABLE_PDM, HIGH); // Disable PDM clock
    Serial.println("setup for IMU done");
#else
    digitalWrite(ENABLE_PDM, LOW); // Enable PDM clock
    Serial.println("setup for audio done");
#endif

    timer4.enable(true); // enable 1mS timer interrupt

    startingFWAddress = indirectRead(0x1fffc0c0);

    ei_setup();
}

2. 主循环

主循环实际上在syntiant_loop()里完成。

void loop(void)
{
    syntiant_loop();
}

void syntiant_loop(void)
{
    // Loop to stay in Standby Mode unless we get a ": " from USB
    // OR interrupt from NDP
    while (1)
    {

        if (flashflag == 1)
        {
            if (++flashcnt >= 500)
            {
                flashcnt = 0;
                digitalWrite(LED_GREEN, !digitalRead(LED_GREEN));
            }
        }
        else
        {
            flashcnt = 0;
        }

        if (match)
        {
            processMatch();
        }
        if (timer4TimedOut)
        {
            timer4TimedOut = 0;
            // Turn Off Arduino LED
            digitalWrite(LED_RED, LOW);
            /*digitalWrite(LED_BLUE, LOW);
            digitalWrite(LED_GREEN, LOW);*/
        }
        // check USB serial port for ": " command from host
        // if (Serial.read() == ':')
        // {
        //     break;
        // }
        if(ei_command_line_handle() == true) {
            break;
        }

        // Deep sleep only if USB disconnected.
        SCB->SCR &= !SCB_SCR_SLEEPDEEP_Msk; // remove deep sleep bit

        if ((USB->DEVICE.STATUS.reg & 0xc0) != idle)
        {
            // we are connected to USB
            USBConnected = 0;
            timer4.enableInterrupt(true); // enable interrupt for Audio
            if (detached == 1)
            {
                detached = 0; // assume attached to host USB
                Serial2.println("Recommected to USB");
            }
        }
        else
            USBConnected += 1;

        if (USBConnected > 0xfff00)
        {
            USBConnected = 0;

            // Only deep sleep (Standby) if LED timer has expired.
            // See if LED timer = 0, & timed out flag not set
            if ((!ledTimerCount) && (!timer4TimedOut))
            {
                SCB->SCR |= SCB_SCR_SLEEPDEEP_Msk; // enable Deep Sleep

                // This saves 0.63mA when battery powered. But then needs
                // keyword interrupt to wake CPU up
                USBDevice.detach();
                detached = 1;

                // stop timer as we are going to deep sleep
                timer4.enableInterrupt(false);

                // Put Flash into Deep Power Down
                digitalWrite(FLASH_CS, LOW);
                SPI1.transfer(FLASH_DP); // enable RESET
                digitalWrite(FLASH_CS, HIGH);
                delayMicroseconds(1);
                delay(1);

                delay(1000);
                Serial2.println("Deep Sleep");
                delay(2000);
            }
            else
            {
                timer4.enableInterrupt(true);
            }
        }

        digitalWrite(1, HIGH);

        __DSB(); // Data sync to ensure outgoing memory accesses complete
        __WFI();
        if (REG_SYSCTRL_DFLLMUL != SAVE_REG_SYSCTRL_DFLLMUL)
        {
            REG_SYSCTRL_DFLLMUL = SYSCTRL_DFLLMUL_MUL(0xBB80) | SYSCTRL_DFLLMUL_FSTEP(1) | SYSCTRL_DFLLMUL_CSTEP(1);
        }

        digitalWrite(1, LOW);

        // Woken up. See if attached to USB host
        if (detached)
        {
            USBDevice.attach();
            Serial.begin(115200);
            detached = 0;
        }
    }
    // Stop timer4 as we will access NDP from main()
    timer4.enableInterrupt(false); // disable 1mS timer interrupt

    // ':' received -- perform a management command
    runManagementCommand();

    timer4.enableInterrupt(true); // enable 1mS timer interrupt
}

3. 模型输出

在定时器4中断里处理NDP101的所有反馈。

NDP.poll()返回匹配的分类match，然后传给模型输出函数ei_classification_output()，因为类标签用字符串数组储存，所以传给的是match-1。

// Timer 4 interrupt. Handles ALL touches of NDP. Also services USB Audio
void isrTimer4(struct tc_module *const module_inst)
{
    digitalWrite(0, LOW);
    int s;
    unsigned int len;

    SCB->SCR &= !SCB_SCR_SLEEPDEEP_Msk; // Don't Allow Deep Sleep
    if ((ledTimerCount < 0xffff) && (ledTimerCount > 0))
    {
        ledTimerCount--;
        if (ledTimerCount == 0)
        {
            timer4TimedOut = 1;
        }
    }


#ifdef WITH_AUDIO
    if (runningFromFlash) {

        uint32_t tankRead;
        int i;

        for (i = 0; i < 32; i += 4)
        {
            tankRead =
                indirectRead(tankAddress + ((currentPointer - 32 + i) % tankSize));
            audioBuf[i / 2] = tankRead & 0xffff;
            audioBuf[(i / 2) + 1] = (tankRead >> 16) & 0xffff;
        }
        currentPointer += 32;
        currentPointer %= tankSize;
    }
    AudioUSB.write(audioBuf, 32); // write samples to AudioUSB
#else

    if(runningFromFlash) {
        currentPointer = indirectRead(startingFWAddress);

        int32_t diffPointer = ((int32_t)currentPointer - prevPointer);

        if(diffPointer < 0) {
            diffPointer = (64000 - prevPointer) + currentPointer;
        }
        if(diffPointer >= dataLengthToBeSaved) {
            len = sizeof(dataBuf);
            int ret = NDP.extractData((uint8_t *)dataBuf, &len);

            if(ret != SYNTIANT_NDP_ERROR_NONE) {
                ei_printf("Extracting data failed with error : %d\r\n", ret);
            }
            else {
                if(imu_active == false) {
                    imu_active = true;
                    for(int i = 0; i < (dataLengthToBeSaved / 2); i++) {
                        imu[i] = dataBuf[i];
                    }

                    imu_active = false;
                }
            }

            prevPointer = currentPointer;
        }
    }
#endif

    if (doInt)
    {
        doInt = 0;
        // Poll NDP for cause of interrupt (if running from flash)
        if (runningFromFlash)
        {
            match = NDP.poll();

            if (match)
            {
                // Light Arduino LED
                //digitalWrite(LED_BUILTIN, HIGH);

                ei_classification_output(match -1);

                printBattery(); // Print current battery level

            }
        }
        else
        {
            match = 1;
        }
    }
    digitalWrite(0, HIGH);
}

ei_classification_output()函数用于输出模型的识别结果，以字符串形式返回标签名给on_classification_changed()函数。标签名存储在model_variables.h中。

void ei_classification_output(int matched_feature)
{
    if (ei_run_impulse_active()) {

        ei_printf("\nPredictions:\r\n");

        for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
            ei_printf("    %s: \t%d\r\n", ei_classifier_inferencing_categories[ix],
                (matched_feature == ix) ? 1 : 0);
        }

        on_classification_changed(ei_classifier_inferencing_categories[matched_feature], 0, 0);
    }
}

4. 功能实现

void on_classification_changed(const char *event, float confidence, float anomaly_score) {

    // here you can write application code, e.g. to toggle LEDs based on keywords
    if (strcmp(event, "openlight") == 0) {
        // Toggle LED
        digitalWrite(LED_GREEN, HIGH);
    }

    if (strcmp(event, "closelight") == 0) {
        // Toggle LED
        digitalWrite(LED_GREEN, LOW);
        flashflag = 0;
    }

    if (strcmp(event, "lightflash") == 0) {
        flashflag = 1;
    }
}

if (flashflag == 1)
        {
            if (++flashcnt >= 500)
            {
                flashcnt = 0;
                digitalWrite(LED_GREEN, !digitalRead(LED_GREEN));
            }
        }
        else
        {
            flashcnt = 0;
        }

这里控制的是RGB灯的绿灯。

在定时器4的中断函数中对模型的输出进行了处理，模型输出的函数ei_classification_output()在中断里被调用，然后将识别结果传给on_classification_changed()函数。

然后在on_classification_changed中检测传来的标签名，若识别到标签openlight（对应指令“打开灯”）就打开灯；若识别到标签closelight（对应指令“关闭灯”）就关灯；若识别到标签lightflash（对应指令“灯闪烁”）就让标志flashflag置1。在主循环中，若标志位flashflag为1，则执行灯闪烁的代码，否则清空闪烁计数器flashcnt。

 

效果演示

“打开灯”

“关闭灯”

“灯闪烁”

 

 

心得体会

本次是第一次参加Funpack活动，通过活动了解到了机器学习的相关知识，有不少收获。

也是第一次接触Arduino相关的东西，Arduino早就听说过，虽然这块板子并非Arduino官方硬件，但是这次Adruino IDE的使用体验并不是太好，有时候总会出现一些奇怪的错误。

不过总的来说，还是完成了这次活动，也感谢硬禾学堂和得捷提供这次机会，希望以后有更多的活动可以参与。