HI there, got an odd hanging issue

Post here first, or if you can't find a relevant section!
Post Reply
lukew
Posts: 7
Joined: Sat Feb 22, 2020 4:05 pm
Answers: 1

HI there, got an odd hanging issue

Post by lukew »

I'm having issues with FreeRTOS on a Bluepill locking up.

I'm working on a state machine and it runs inside a timer event once every 5 seconds. If I use Serial.print() from within a switch statement, the thing locks up and the LED blinks 3 times, then stays solid for 2 seconds and repeats. If I uncomment another serial print in the same function but before the swtich, it works fine.

It also does similar within an LCD update routine that runs every second. I added an if/else statement to it to either display a setpoint for the control process, or "OFF" depending on the program state. Now the LCD works initially by displaying startup data and FW version, but then after the code blanks it, it jams up with the same LED blinking.

I have been adding serial prints as debug messages so I can see where the code stops working, however if I put one after the display blanking code, it starts working fine. Remove it and it locks up. Also adding a delay does the same and the code works fine.

Any ideas on what may be causing this? The project is a programmable process controller so testing every single combination of conditions the program has to deal with is impossible.
ag123
Posts: 1655
Joined: Thu Dec 19, 2019 5:30 am
Answers: 24

Re: HI there, got an odd hanging issue

Post by ag123 »

one of those things about a blue pill (rather stm32f103c8) is it has 20k sram for 'everything'.
and if you run a context switching RTOS, it needs to partition a smaller part of the memory into smaller silos for the stack and some variable for each vtask. so in no time you would run out of memory. so you can try to reduce the number of vtask slots to reduce memory use

the other thing is that things like Serial print may not be re-entrant, usually it doesn't lead to a lockup more than that it may overwrite some data.
but it can lead to a lockup if part of that code use a spin lock e.g.

Code: Select all

while(1) 
  if(check_variable()) break;
if you have 2 threads writing to the same variable, one of the thread would exit that while loop. the other gets stuck in that while loop and never exits.

there is a bit of discussion about RTOS here:
viewtopic.php?f=24&t=166

there has been once i got so bothered about RTOS and its memory fragmentation, i got out of RTOS and resort to using an event loop.
thereafter, 'everything works' and i never looked back.
take a look at the thread mention above, Even an event loop is not really necessary, one can simply call the functions in loop() and just keep state and do not hog time slices in a spin lock
lukew
Posts: 7
Joined: Sat Feb 22, 2020 4:05 pm
Answers: 1

Re: HI there, got an odd hanging issue

Post by lukew »

I did think about memory, I had it print out the result of xPortGetFreeHeapSize and it was about 10k. All of the program variables are stored in a single global struct with each task only needing a couple of int's to work with.

I do have 2 devices sharing an SPI port but I use a semaphore to prevent them from interfering with each other, they just give up if they can't take it as nothing is timing critical.

No 2 tasks will ever try to write the same variable, they can read them but not write. I'd have thought that the OS would keep running if a thread gets stuck in a loop. The LED that blinks is a relay output and nothing in my code will make it blink the way it does, I just used the LED pin as a visual indicator (the relay is a photovoltaic).
ag123
Posts: 1655
Joined: Thu Dec 19, 2019 5:30 am
Answers: 24

Re: HI there, got an odd hanging issue

Post by ag123 »

Serial.print() in particular USB serial (and maybe even hardware serial (i.e. uart)) may use a queue or circular buffer.
that queue / circular buffer has the same implications as writing to shared variables. they use shared variables and the queue, circular buffer itself is shared.
while this is outright speculation, i.e. it need not be related to Serial.print() itself it is one of those things to be wary if 2 vtasks calls Serial.print()
that could also happen with many other registers etc that requires a spin lock. e.g. to check state for readiness and writing to those registers.
the h/w registers works just like variables, so that if two threads work on/write to them, one may be caught in a spinlock that never exits

and there is still the problem about RTOS and memory fragmentation. you may like to try to configure the number of vtasks, memory etc to see if reducing memory foot print helps

as for the non-re-entrant functions in the core, you may need to use semaphores or mutex for particular ones that spinlock around variables or hardware registers. nearly 'everything'. so that it becomes one thread at a time. so it may not be easy to solve
ag123
Posts: 1655
Joined: Thu Dec 19, 2019 5:30 am
Answers: 24

Re: HI there, got an odd hanging issue

Post by ag123 »

btw another hint though, in stm32(f103) and other 'single core' ones, there is really only one thread, it is either running in thread (main program) mode or in an interrupt handler. so one way to figure things out is to use the debugger. to figure out where it got stuck. knowing that it is single threaded probably makes debugging somewhat easier.

when i hit that, i didn't go with debug, i tried to reduce memory usage, 20k + that rtos fragmentation is really very tight. just imagine that the os and the rest use 15k of memory and leaves 5k for your processes and tasks. well, it does not really do that but say
the global stack is say 3k, then say there are 4 vtasks and say you allocate 2k to each vtask. that already consume 3k + 4 x 2k = 11k just for the stacks.
then there is still need for space for global variables lets say 1k, then buffers for each of usb-serial, uart, spi etc etc. 1st global memory likely runs out and there is no memory exception, the stack simply overwrite global variables if it overflow. and each vtask say has 2k stack and the global stack say 3k if any one of those stacks overflows, it overwrites something else on another stack or some other global variables silently. so hangs is pretty much 'expected'
lukew
Posts: 7
Joined: Sat Feb 22, 2020 4:05 pm
Answers: 1

Re: HI there, got an odd hanging issue

Post by lukew »

Not got a clue what was up with it but after updating the core from 1.7 to 1.8, relaunching the IDE and adding come more control logic to the control loop, it started workng fine.

It's just completed a simulated control cycle and performed perfectly fine. The LED was indicating something but what I can't find out.
mrburnette
Posts: 633
Joined: Thu Dec 19, 2019 1:23 am
Answers: 7

Re: HI there, got an odd hanging issue

Post by mrburnette »

lukew,

Your experience is not unusual. freeRTOS is a great product, I use it integrated in the Arduino for ESP32. When things work as expected, everyone cheers. There are tools that may help, but it is not free! I personally object to paying for diagnostics for free/open source add-in library code. For most end-users the trace hooks are adequate.

I would recommend taking the trace macros code and implementing it into Arduino format. Build out the sketch to do a variety of things: blink, count, serial print, etc. Get this sketch working perfectly. Now when you run into a freeRTOS issue, take a copy of your working sketch and insert questionable code into the working model until you identify the issue.

Good luck,

Ray

ESP32 example: here
lukew
Posts: 7
Joined: Sat Feb 22, 2020 4:05 pm
Answers: 1

Re: HI there, got an odd hanging issue

Post by lukew »

Thanks for the advice. I've just found out that I likely have boards with the fake micros on them so until I get some PCB's made with genuine STM parts on them I'll hold off on any bug finding.
ag123
Posts: 1655
Joined: Thu Dec 19, 2019 5:30 am
Answers: 24

Re: HI there, got an odd hanging issue

Post by ag123 »

there is something we often forget when we do bare metal programming, there is no OS, the code that you are writing is part of the os, you are extending it in setup() and loop(), spin locks, direct h/w registers write is the norm. unlike 'big' os where there is memory protection, admins/root etc. on a stm32 mostly you are admin/root, and your code is the OS itself, no layers, no nothing between, pure hardware and stm32 with its rich features amplifies that.
Post Reply

Return to “General discussion”