Modified IotaWatt is stuck on perpetual watchdog reset

Hi,

I have an IotaWatt and I have been trying to modify the code to increase the sampling rate from 5 samples/sec to 1 sample/sec. After maybe the 12 reflashing of the firmware, the unit now just resets perpetually.

Things I have tried:

  • reflash with original GIT hub firmware
  • pull battery to reset rtc
  • tried a powered USB hub (I checked the forums for this, really?! a power problem causes a WDT reset? In desperation I tried it anyway.
  • tried to format SDCard but it is locked.

Anyone know how to recover this? Is this not a firmware issue and is it really a hardware issue? See below for the serial ouput from the device.

Serial Initialized
No clock yet: SPI started.
No clock yet: SD initialized.
No clock yet: Real Time Clock not initialized.
No clock yet: Version: 02_02_27
No clock yet: Reset reason: Software Watchdog
No clock yet: Trace: 130,131,132,133,134,135,136,137,138,139,11,90,12,13,14,15,30,16,17,18,11,90,91,92,12,13,14,15,25893,11074,9413,1974
No clock yet: ESP8266 ChipID:1852989
No clock yet: device name: IotaWatt, version: 3
No clock yet: Local time zone: -5
No clock yet: MDNS responder started
No clock yet: You can now connect to http://IotaWatt.local
No clock yet: HTTP server started
No clock yet: dataLog: service started.

Soft WDT reset

          ctx: cont
                    sp: 3fff3d60 end: 3fff4070 offset: 01b0

stack>>>
3fff3f10: 3fff72b4 00000100 3fff5d34 4020fe2c
3fff3f20: 00000100 00000000 00009146 402107ec
3fff3f30: 3fff0b80 00000008 3fff5d34 5a3c0771
3fff3f40: 5a48186d 3fff72b4 3fff2ac0 4020f004
3fff3f50: 3fff4a08 00000000 3fff3fb0 40213f41
3fff3f60: 3fff4a08 3fff2ac0 3fff72b4 40213ffa
3fff3f70: 5a481870 0000410c 00000000 40214515
3fff3f80: 3ffeacf8 00000000 000003e8 00000000
3fff3f90: 5a48186d 3fff72b4 3fff2ac0 40214184
3fff3fa0: 0000410c 00000000 00000000 00000000
3fff3fb0: 00000005 00000000 00000000 40100688
3fff3fc0: 00000548 00000000 3fff2cc8 3fff72b4
3fff3fd0: 00000548 3fff2ac0 3fff2cc8 40215f5f
3fff3fe0: 00000000 00000000 00000000 40202064
3fff3ff0: 00000001 3fff2f60 3fff3050 3fff2f60
3fff4000: 3fff27bc 3ffe8550 40202eb8 3fff3050
3fff4010: 00000000 00000000 83aa7e80 3ffe8550
3fff4020: 3fff27b8 3fff2f60 3fff6634 4021522f
3fff4030: 0000000f 00000000 feefeffe feefeffe
3fff4040: 00000000 00000000 00000001 3fff3040
3fff4050: 3fffdad0 00000000 3fff3039 40202f04
3fff4060: feefeffe feefeffe 3fff3050 402042a4
<<<stack<<<

ets Jan 8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1384, room 16
tail 8
chksum 0x2d
csum 0x2d
v00000000
~ld

** Restart **

Serial Initialized
No clock yet: SPI started.
No clock yet: SD initialized.
No clock yet: Real Time Clock not initialized.
No clock yet: Version: 02_02_27
No clock yet: Reset reason: Software Watchdog
No clock yet: Trace: 130,131,132,133,134,135,136,137,138,139,11,90,12,13,14,15,30,16,17,18,11,90,91,92,12,13,14,15,25893,11074,9413,1974
No clock yet: ESP8266 ChipID:1852989
No clock yet: device name: IotaWatt, version: 3
No clock yet: Local time zone: -5
No clock yet: MDNS responder started
No clock yet: You can now connect to http://IotaWatt.local
No clock yet: HTTP server started

I have no idea what this means. There are so many things that can potentially get screwed up I don’t know where to begin.

You need to start over. Completely reformat your SDcard or get a new one, then load it with just what’s in the SD directory on github. Load the firmware with the binary from release 02_02_27. The IoTaWatt should work.

Be advised I won’t support attempts to change the fundamental architecture of the firmware. Making modifications like this require that you understand how it all works and that you have the skills to debug.

Good luck.

A post about sample rates

Notwithstanding what Bob has said… The ESP watchdog will reset the system if you spend too long in a loop which I’m guessing you have inadvertently done with your code. There are ways round this. If you search one of the ESP forums or read some of the tutorials you’ll find the information.

Simon

Thanks for the quick response. Simon, that would mean that the GIT hub sources are broken. I have compiled fresh sources with none of my changes and I am getting this error. My changes, unless stored somewhere in NVM and are not being overwritten with a re-flash, should return the board to “factory defaults.”

Either the issue lies in the SDCard and should disappear with a reformat, or this is extremely sensitive hardware. It does not make any sense to me why the board would have accepted nearly a dozen reflashes and suddenly has a perpetual reset issue.

Question: I tried to format the sdcard on my pc. I could not make any changes to the sdcard because it is locked. 1) is the card supposed to be locked? 2) how do you unlock a microsd that does not have a lock switch on it?

“how do you unlock a microsd that does not have a lock switch on it?”

you sure your SD card did not simply fail

You can damage the card by removing while the device is running. ioTaWatt doesn’t lock the card. You would do well to get a new card and load it as previously stated. Also, it would be best if you flashed the binary on github rather than compile your own. Use esptool to load.

Possibly. Sdcard are known to crap out randomly, but I could read the files on my pc. When I tried to delete the, my pc said the card was locked.

I never removed the sdcard while the system was powered. I may have cycled power while it was writing, but SDCard failure doesn’t line up with the debug output. Sound advice on using GIT binaries. I shall try this and see If I can get it to recover.

Bob, if you care to take a look at this issue I ran into, please share with me your email, I would like to send you some log files and point some places where I see some areas of concern.

I was not able to recover the SDCard that came with unit. It is forever locked, even if I follow these instructions: My Disk is Read-only, Help! | Microsoft Docs

Not sure if you are intentionally locking the card, but this maybe the reason why the IotaWatt is rebooting. I am thinking the unit is trying to write to a readonly SDCard and the watchdog is waking up because the writes are unsuccessful.

I used a brand new SDCard and dropped the requisite base files on the card and the unit started working again. I am able to get 1 second sampling resolution with code my modification.

At first I thought this might be data size issue with the card getting full or reaching the FAT32 file system limits, but it’s not. On one Iotawatt, the log file is only 4MB.

One the second Iotawatt that started resetting, the log file was only 400KB.

Can you confirm that the firmware is not locking the SDCard to be read only?

I will run my code overnight with the new SDCard to see if I reproduce the problem on an new unlocked card (16GB Sandisk)

Mieszko, it appears that you were able to get your unit running with a new SDcard and unmodified firmware. There are no incidences of SDcard problems being reported with the standard firmware. You have the sources and are free to modify to suit your needs, but I’ve got too much to do already and won’t get involved.

Perhaps if you hook up with some of the folks in the home brew thread, there are some folks there who have demonstrated a deeper understanding of the firmware and might be willing to help you debug your project.

Good luck.

Hi Bob,

Just to let you know, the problem reoccurred with the new card and your released firmware. Looking at the logs there look to be power failures and external resets happening every 2-3 minutes for an hour or two, which I cannot correlate to anything happening on the power in my house. I have tried several different supplies and batteries and this problem persists. Even with power interruptions, your product is technically supposed to recover under EN61000-4 without operator intervention (CE certification that your product declares).

Additionally, after these anomalies, the real-time clock time-warps to the future. I am writing this at 10:45 am on 1/7/2018. The unit thinks it is in the future or possibly in Dubai or something.

I will have to revise my original theory of the Log files being corrupted or read-only being magically turned on the SDCard. I cannot explain it, nor do I have any more time to investigate these issues any further, but the removal of the log files did not recover the unit. The real trick was to replace all the files in the root directory, config.txt etc. It seems like these files might becoming corrupted.

Unfortunately, I will not be able to recommend this product as an option for installation. Thanks for your time and I understand your inability to support with these issues. Best of luck with your product.

OK Mieszko. So just to recap:

  1. You unsuccessfully modified the firmware to do something it wasn’t designed to do. You wanted me to debug your effort and I declined to do that.

  2. I suggested you get a new SDcard, load it with an image from github, and load the official binary release firmware.

  3. Your last post indicated the device was working, but that your modified firmware was not.

  4. Now that I won’t help you debug your software, you are saying the device isn’t working properly because it gets power failures every 2-3 minutes and the RTC is malfunctioning.

The log you posted shows that the device had been working for 21 minutes before the power failure. Also, if you examine the log further, you will notice that the timestamp on restart is UTC, and is adjusted later during the boot process for the local time difference that is contained in the configuration file. The inconsistency between your description of the problem and what is in the log only reinforces my reluctance to get involved in projects like this.

Look, there are a lot of people on this forum that are happy to talk all day about modifying these products to do all kinds of things. I’m just not one of them. I’ve got a long list of enhancements that I want to get done on this product for the benefit of all users. What you want me to do is a rabbit hole I’ve been down before.

Power failures are an external issue. If you want me to verify that, post the msglog file - all of it, and I’ll take a look at what you have been up to and see if it looks like a defect. I doubt it.