Corrupt CSV files being logged

Post Reply
User avatar
intellact
Forum Czar
Posts: 703
Joined: Mon Nov 28, 2005 3:20 am
Location: Ontario, Canada
Contact:

Corrupt CSV files being logged

Post by intellact » Sun Nov 08, 2015 8:15 pm

So I finally have the AXS Port device up and running. (Which means that all WattPlot applications will soon be AXS-compatible! \:D/ )

But first things first...

The AXS Port, like the MATE3, can log its own CSV data files. However, have you ever tried to open one of those files in Notepad? You can't see anything, because these text files are riddled with NULs. :shock: Those are ASCII 00 chars, normally used to terminate strings. They should never appear inside a CSV file. It looks like they are being used as a delimiter. :mad: That's just wrong.

I reported a similar problem with the MATE3 SD files which was fixed in a later firmware release. I wonder if this will be fixed for the AXS Port as well...?
Andrew Welch, creator of WattPlot

LorenAmelang
Forum Whiz
Posts: 37
Joined: Mon Oct 26, 2009 10:51 pm
My RE system: (being majorly revised...)

Re: Corrupt CSV files being logged

Post by LorenAmelang » Tue Dec 22, 2015 9:59 pm

Totally agree about the Nulls! But in my AXS there are other problems as well...

I'm trying to log once per minute, but several times each hour a minute or two at a time are just skipped.

Every now and then the uSD just disconnects somehow, leaving a corrupted file that can't be read or copied, and a big gap in the logging until I go remove and re-insert the card. I just installed a new card, with silicone grease on the contacts, to see if that makes any difference...

Last night while genset charging, I saw a case where the logs, and my (non-OutBack) inverter were saying I was charging at 29.4 V, but both of the Modbus items in the AXS view of the FM80 were showing 27.4 V! I'm pretty sure 29.4 was correct, but I didn't go out there in the pouring rain with a meter. Maybe the MPPT doesn't guarantee good voltage readings unless it is doing the charging?

And finally, I've only found one FTP client that works - the ancient WS-FTP95, which isn't even Y2K compatible. DO NOT DOWNLOAD FileZilla - they are installing the "Chromium" browser hack along with it! See my next post...

Loren

User avatar
intellact
Forum Czar
Posts: 703
Joined: Mon Nov 28, 2005 3:20 am
Location: Ontario, Canada
Contact:

Re: Corrupt CSV files being logged

Post by intellact » Thu Feb 04, 2016 8:23 am

LorenAmelang wrote:DO NOT DOWNLOAD FileZilla - they are installing the "Chromium" browser hack along with it!
Might you have used SourceForge to download your FileZilla? That seems to be a major contributor to problems. I like FileZilla myself.

But back to the main thread...

I have now seen several instances where CSV data logs seem to 'run on' into other logs. E.g. the log runs from hh:00:00 to hh:59:59, and then you get large chunks of what appear to be completely different logs. :mad: No doubt the NULs are a contributing factor.

I am sick and tired of having to put in huge kludgy workarounds into WattPlot just to try and deal with OutBack's bad data. "Software errors" I can live with (I make them all the time :wink:) - but it is this refusal to fix glaring errors after they have been pointed out that really gets me steamed. ](*,)

Perhaps one day their firmware updates will have the guts to say what was fixed, so we know what's still a problem - or what used to be. :roll:
Andrew Welch, creator of WattPlot

LorenAmelang
Forum Whiz
Posts: 37
Joined: Mon Oct 26, 2009 10:51 pm
My RE system: (being majorly revised...)

Re: Corrupt CSV files being logged

Post by LorenAmelang » Thu Feb 04, 2016 1:34 pm

Andrew,

> I have now seen several instances where CSV data logs seem to 'run on' into other logs. E.g. the log runs from hh:00:00 to hh:59:59, and then you get large chunks of what appear to be completely different logs.

I see this when the AXS internal time "jumps" around. If you don't manually enable NTP after each power cycle of the AXS Port, and the time creeps out of sync beyond some unknown amount (I think it has to do with the midnight reset of logging from the AXS versus the end of charging day reset from my FM80) the internal timestamps will jump in eight hour segments. But there are two separate internal times, one for the log file names, and one for the timestamps inside the files. So you will see some filenames re-used for additional log lines from different timestamp times.

I can't "verify" my AXS MAC for Optics (I can interact with my neighbor's system fine), so I can't reach the settings to reduce the data rate it uploads - which would triple my monthly data charges if I didn't block it in my router. Which means I have to manually open the block for a few minutes every week or so to let it fix its internal time, to avoid this log file chaos.


> Perhaps one day their firmware updates will have the guts to say what was fixed, so we know what's still a problem - or what used to be.

This is copied verbatim from the OutBack reply - I take it this is not fixed:
-----
For the twelfth time, could someone please answer this question?

>> And... Would it be possible to get an answer whether the password
>> bug that bricked my original AXS Port has ever been fixed? If I
>> accidentally hit the 'u' key while using the Shell program, am I
>> going to destroy this new hardware? If I do hit the 'u' key, is there
>> any way to break out of the "Enter Password" process without setting it?

The response from engineering is to use Ctl-C.
-----

User avatar
intellact
Forum Czar
Posts: 703
Joined: Mon Nov 28, 2005 3:20 am
Location: Ontario, Canada
Contact:

Re: Corrupt CSV files being logged

Post by intellact » Thu Feb 04, 2016 5:09 pm

LorenAmelang wrote:I see this when the AXS internal time "jumps" around. If you don't manually enable NTP after each power cycle of the AXS Port, and the time creeps out of sync beyond some unknown amount (I think it has to do with the midnight reset of logging from the AXS versus the end of charging day reset from my FM80) the internal timestamps will jump in eight hour segments. But there are two separate internal times, one for the log file names, and one for the timestamps inside the files. So you will see some filenames re-used for additional log lines from different timestamp times.
Yeah - endless fun there. :roll:
LorenAmelang wrote:I can't "verify" my AXS MAC for Optics (I can interact with my neighbor's system fine), so I can't reach the settings to reduce the data rate it uploads - which would triple my monthly data charges if I didn't block it in my router. Which means I have to manually open the block for a few minutes every week or so to let it fix its internal time, to avoid this log file chaos.

The AXS-compatible WattPlot 5.1 allows a user-specified polling frequency, so it will just grab the system state every 1, 2, 3, 5, 10, 20, 30, or 60 seconds. :smile: It also back-fills data gaps so that the power summaries are as precise as the frequency.

Furthermore, it has the option of logging all incoming data in the AXS/MATE3 CSV file format, although my hourly files are boringly just hourly, with times that are actually sequential. :-"
Andrew Welch, creator of WattPlot

Post Reply