Erroneous and inconsistent LMIC_PRINTF_TO behavior

@tmm

I ran into a strange issue using LMIC-node and the MCCI LMIC library:

Part 1:

In LMIC-node I use a reference named serial that to points to the correct serial port (Serial or SerialUSB) which is hardware dependent, so I can write to serial independently of the hardware used.

Without thinking too much I did set #define LMIC_PRINTF_TO serial in my application code and it ‘just’ worked (somehow…) so I didn’t bother about it actually. I did notice that it did not work for situations where SerialUSB is used instead of Serial.

A user posted an issue that this could not work because when LMIC gets compiled it is not aware of the LMIC_PRINTF_TO serial in my application code.
This is correct so when I don’t set LMIC_PRINTF_TO anywhere at all, and enable debug output (in platformio.ini: -D LMIC_DEBUG_LEVEL=1 then I should not see any debug output arriving in the serial monitor - UNLESS LMIC uses Serial as default when no LMIC_PRINTF_TO is defined by the user.

I am using Windows 10 and it appears that LMIC uses Serial by default for debug output.
(Which is why I initially thought that setting LMIC_PRINTF_TO in my application code worked, but actually it does not).

The user who mentioned the issue is using Mac OS and an Adafruit Feather M0 LoRa board (and region au915 instead of eu868). According to his/her feedback (s)he could only get debug output to the serial monitor working when adding -D LMIC_PRINTF_TO=Serial to platformio.ini. This means that the behavior on his/her system is different: Serial is not used as default when LMIC_PRINTF_TO is not specified (but it does on my Windows system).

Part 2:

I noticed that debug output was not working for two boards. Probably because they use/need SerialUSB instead of Serial. These boards are Raspberry Pi Pico and SAMD21 M0-Mini (‘Arduino Zero (USB)’).

When I define -D LMIC_PRINTF_TO=SerialUSB in platformio.ini and compile for these boards, I get the following compile errors:

Compiling .pio\build\zerousb\lib069\MCCI LoRaWAN LMIC library\lmic\lmic_as923.c.o
Compiling .pio\build\zerousb\lib069\MCCI LoRaWAN LMIC library\lmic\lmic_au915.c.o
Compiling .pio\build\zerousb\lib069\MCCI LoRaWAN LMIC library\lmic\lmic_compliance.c.o
Compiling .pio\build\zerousb\lib069\MCCI LoRaWAN LMIC library\lmic\lmic_eu868.c.o
.pio\libdeps\zerousb\MCCI LoRaWAN LMIC library\src\hal\hal.cpp:380:8: error: 'cookie_io_functions_t' does not name a type
 static cookie_io_functions_t functions =
        ^~~~~~~~~~~~~~~~~~~~~
.pio\libdeps\zerousb\MCCI LoRaWAN LMIC library\src\hal\hal.cpp: In function 'void hal_printf_init()':
.pio\libdeps\zerousb\MCCI LoRaWAN LMIC library\src\hal\hal.cpp:389:37: error: 'functions' was not declared in this scope     stdout = fopencookie(NULL, "w", functions);
                                     ^~~~~~~~~
.pio\libdeps\zerousb\MCCI LoRaWAN LMIC library\src\hal\hal.cpp:389:37: note: suggested alternative: 'union'
     stdout = fopencookie(NULL, "w", functions);
                                     ^~~~~~~~~
                                     union
.pio\libdeps\zerousb\MCCI LoRaWAN LMIC library\src\hal\hal.cpp:389:14: error: 'fopencookie' was not declared in this scope
     stdout = fopencookie(NULL, "w", functions);
              ^~~~~~~~~~~
.pio\libdeps\zerousb\MCCI LoRaWAN LMIC library\src\hal\hal.cpp: At global scope:
.pio\libdeps\zerousb\MCCI LoRaWAN LMIC library\src\hal\hal.cpp:376:16: warning: 'ssize_t uart_putchar(void*, const char*, size_t)' defined but not used [-Wunused-function]
 static ssize_t uart_putchar (void *, const char *buf, size_t len) {
                ^~~~~~~~~~~~
*** [.pio\build\zerousb\lib069\MCCI LoRaWAN LMIC library\hal\hal.cpp.o] Error 1

So type cookie_io_functions_t is unknown.

My next test was to explicitly define -D LMIC_PRINTF_TO=Serial (or SerialUSB) for each board individually and compile for all boards.

To my surprise, for some boards that use Serial, I got the same compile errors as above with the boards using SerialUSB: cookie_io_functions_t is unknown.

But when not defining LMIC_PRINTF_TO it was (miraculously) possible to get LMIC debug output to the serial monitor for the same boards, while when defining -D LMIC_PRINTF_TO=Serial causes above compile errors.

Below are the results of compiling the code for all boards:

Environment                 Status    Duration
--------------------------  --------  ------------
adafruit_feather_m0_lora    FAILED    00:00:15.132
disco_l072cz_lrwan1         FAILED    00:00:45.243
heltec_wifi_lora_32_v2      SUCCESS   00:00:50.371
heltec_wifi_lora_32         SUCCESS   00:00:50.124
heltec_wireless_stick_lite  SUCCESS   00:00:49.304
heltec_wireless_stick       SUCCESS   00:00:49.885
lopy4                       SUCCESS   00:00:50.160
lora32u4II                  SUCCESS   00:00:16.223
ttgo_lora32_v1              SUCCESS   00:00:49.618
ttgo_lora32_v2              SUCCESS   00:00:49.425
ttgo_lora32_v21             SUCCESS   00:00:49.166
ttgo_t_beam                 SUCCESS   00:00:46.935
ttgo_t_beam_v1              SUCCESS   00:00:50.734
blackpill_f103c8_128k       FAILED    00:00:41.381
blackpill_f103c8            FAILED    00:00:42.120
bluepill_f103c8_128k        FAILED    00:00:42.346
bluepill_f103c8             FAILED    00:00:42.122
lolin_d32_pro               SUCCESS   00:00:50.027
lolin_d32                   SUCCESS   00:00:49.545
lolin32                     SUCCESS   00:00:49.258
nodemcu_32s                 SUCCESS   00:00:49.516
nodemcuv2                   SUCCESS   00:00:28.387
pico                        FAILED    00:00:32.191
pro8mhzatmega328            SUCCESS   00:00:15.211
samd21_m0_mini              FAILED    00:00:06.158
teensylc                    FAILED    00:00:32.872
===== 9 failed, 17 succeeded in 00:17:33.452 =====

(Note: in the repo on Github samd21_m0_mini is still called zerousb. The results above are from a newer version where this board has been renamed.)

It is remarkable that the problems only occurs for ARM related boards.
The compile errors do not occur for AVR, ESP32 and ESP8266 boards.

I tried to understand the LMIC_PRINTF_TO related code in the MCCI LMIC source (hal/hal.cpp) but I do not (yet) fully understand it.

I did notice the following in halp/hal.cpp though:

#if defined(LMIC_PRINTF_TO)
#if !defined(__AVR)

...

#else // defined(__AVR)

...

#endif // !defined(ESP8266) || defined(ESP31B) || defined(ESP32)
#endif // defined(LMIC_PRINTF_TO)

!__AVR in above code appears to mean ESP8266 || ESP32.
(The !defined(ESP8266... remark appears to be a leftover that was not removed).

From this I assume that the code is intended for (and was only tested for?) AVR, ESP8266 and ESP32, which excludes all ARM variants like STM32, SAMD21, RP2040 and the MKL26Z64VFT4 on the Teensy.

That assumption matches my compilation results for each board.

The user that reported the referenced issue uses Mac OS, an Adafruit Feather M0 LoRa board (region au915) and explicitly defines -D LMIC_PRINTF_TO=Serial in platformio.ini and it works for him/her. If I define the same LMIC_PRINTF_TO and compile for the same Adafruit board things don’t work for me and I get the compile errors mentioned above instead.

Questions

  • Does MCCI LMIC use Serial as default for debug output if LMIC_PRINTF_TO is not defined by the user?

  • If not, how is it possible that Serial is used as default on my Windows system?

  • If true, why doesn’t it work for the user that reported the issue and is using Mac OS?

  • Do you have any idea what is going wrong with the compilation errors above?
    Could this be a bug in LMIC or is it caused by something else?

Question noted. Short answer is “printfs are definitely not cleanly implemented”. I’ll give a longer reply later.

I found the cause for the compilation errors and also found a solution. I will provide the details later.

The following took me a quite while to find out:

For debug output / LMIC_PRINTF_TO the LMIC library checks if it is compiled for AVR or not.
If not compiled for AVR it assumes it is compiled for ESP32 or ESP8266.

When not compiled for AVR the code uses certain extensions (e.g. cookie_io_functions_t) that are defined in the ESP32 and ESP8266 Arduino cores.

When compiling for ARM based MCU’s (‘not AVR’) LMIC tries to use the same extensions but these are (by default) not enabled in the ARM compiler. However, these ‘features’ can be enabled by defining the following build parameter: _GNU_SOURCE.

I have now added this in LMIC-node for each (ARM) board separately and my compilation problems on Windows are solved. LMIC debug output now works for all my ARM-based boards (for boards that use Serial and for boards that use Serial USB).
A user has also successfully tested this (LMIC-node v1.3.0) with macOS for his Adafruit Feather M0 LoRa board. So adding the _GNU_SOURCE parameter did not appear to effect things on macOS (where the compilation problem for the ARM based Feather board did not occur).

It would be nice if this solution can get added to the LMIC library, so that users don’t get compilation errors when enabling LMIC debug output for their ARM based board.

I can create a PR for this if you like.

Please see other message: this should all be done as one change. Happy to accept a PR for this.