How to remote download the AXS port’s SD data (csv. file)

How to remote download the AXS port’s SD data (csv. file)

Postby Lawson on Sat Feb 27, 2016 12:21 am

Hi

How to remote download the AXS port’s SD data (csv. file)? I cannot find the way on the OpticsRe dashboard.

Thank you!
Lawson
Member
 
Posts: 16
Joined: Mon Jul 26, 2010 1:52 am
My RE system: SMARTRE 3000-120-NA

Re: How to remote download the AXS port’s SD data (csv. file

Postby intellact on Sat Feb 27, 2016 8:19 am

Forget OpticsRE for something like this.

Pickup an FTP (File Transfer Protocol) tool. You can find a few good ones free on the internet. Personally, I like FileZilla, but I don't recommend getting it from SourceForge - los of complaints about extra software coming with it, etc.

Anyway, once you have an FTP tool, direct it to the IP address of your AXS Port. There you will find a folder called DATALOGS, and in that - voila! - all your CSV files. :grin:

There is a caveat. The files seem to open okay in Excel, but they use ASCII 00 (NULs) as delimiters which is NEVER okay. If you try to open them in MS Notepad, for example, it will grind for a while and you will see nothing. (No acknowledgement yet from OutBack that this is a problem, and so far, no word on when it will be fixed, if ever.)

Also, what you'll get are hourly files of raw data, with the system status every second, which can be difficult to interpret. A tool like WattPlot SumMATE can read the AXS SD Card files, analyse the data, summarize everything into daily statistics, produce plots of activity, and create logs of changes in operating modes, warning, errors, etc, all with a mouse click or two. Check it out. :smile:
Andrew Welch, creator of WattPlot
User avatar
intellact
OutBack Czar
 
Posts: 602
Joined: Mon Nov 28, 2005 4:20 am
Location: Ontario, Canada

Re: How to remote download the AXS port’s SD data (csv. file

Postby Lawson on Sun Feb 28, 2016 9:23 pm

Hi,

Thanks for the information.
Yes, through the FTP I can see the csv files on the DATALOGS files but I cannot download/copy from this files, why?

Lawson
Lawson
Member
 
Posts: 16
Joined: Mon Jul 26, 2010 1:52 am
My RE system: SMARTRE 3000-120-NA

Re: How to remote download the AXS port’s SD data (csv. file

Postby intellact on Mon Feb 29, 2016 7:12 am

Oops! :oops: You're right. For some reason, I had gone so far as to see the CSV files listed under the DATALOGS folder in FileZilla, but I had never actually bothered to try and copy them anywhere.

Sure enough, I got nothing but an uninformative log of failed attempts (see below). :-k Perhaps it is time to forward your question on to OutBack's technical support. Hopefully they will have a better of the FTP procedures they allow (and why). Maybe it is a matter of supplying some kind of password or using a port other than the default (21). :?:

Here's what my log said:

Command: LIST
Response: 150 Opening BINARY mode data connection for '/DATALOGS'.
Response: 226 Transfer complete.

Status: Directory listing successful
Status: Connecting to 192.168.1.5:21...
Status: Connection established, waiting for welcome message...
Error: Connection timed out
Error: Could not connect to server

Status: Delaying connection for 5 seconds due to previously failed connection attempt...
Status: Connecting to 192.168.1.5:21...
Status: Connection established, waiting for welcome message...
Error: Connection timed out
Error: Could not connect to server

Status: Delaying connection for 5 seconds due to previously failed connection attempt...
Status: Connecting to 192.168.1.5:21...
Status: Connection established, waiting for welcome message...
Error: Connection timed out
Error: Could not connect to server
Andrew Welch, creator of WattPlot
User avatar
intellact
OutBack Czar
 
Posts: 602
Joined: Mon Nov 28, 2005 4:20 am
Location: Ontario, Canada

Re: How to remote download the AXS port’s SD data (csv. file

Postby Lawson on Mon Feb 29, 2016 8:26 am

Thank you so much!
Lawson
Member
 
Posts: 16
Joined: Mon Jul 26, 2010 1:52 am
My RE system: SMARTRE 3000-120-NA

Re: How to remote download the AXS port’s SD data (csv. file

Postby LorenAmelang on Fri Mar 04, 2016 2:00 pm

I reported this problem and handed them the solution on January 19. Supposedly it will be fixed in 3.15 - if it ever comes out. They promised me 3.13 by the end of January, and it still isn't available. But they had energy enough to make a "how to install" video...

=====
The reply to LIST is not parsed correctly due to an invalid mask for the files. The mask for every file entry should match the regular expression (((r|-)(w|-)([xsStTL-]))((r|-)(w|-)([xsStTL-]))((r|-)(w|-)([xsStTL-])))\+?\s+.

For example

Code: Select all
  d-------- 1 owner group 0 Dec 22 10:19 DATALOGS should be
  d--------- 1 owner group 0 Dec 22 10:19 DATALOGS
  --------- 1 owner group 1845484 Dec 22 10:20 OPTICS.RPY should be
  ---------- 1 owner group 1845484 Dec 22 10:20 OPTICS.RPY
=====


Sure enough:

AXS:
=====
Code: Select all
LIST
150 Opening ASCII mode data connection for '/'.
d--------   1 owner    group               0 Dec 22 10:19 DATALOGS
---------   1 owner    group         5748760 Dec 22 10:20 OPTICS.RPY
d--------   1 owner    group               0 Dec 22 10:18 SYSTEM~1
=====

Ubuntu:
=====
Code: Select all
-rw-rw-r-- 1 ubuntu ubuntu 154015 Sep 19 14:39 outback.c
-rw-rw-r-- 1 ubuntu ubuntu  45260 Sep 19 14:39 outback.h
-rw-rw-r-- 1 ubuntu ubuntu  15353 Sep 19 14:39 OutbackShell.c
-rwxrwxr-x 1 ubuntu ubuntu 105804 Sep 19 15:03 shell
ubuntu@arm:~/Lpkg/axs/311$
=====

You have one too few items in the first permissions section! Here are the rules:
<https://www-01.ibm.com/support/knowledgecenter/ssw_i5_54/rzaiq/rzaiqrzaiqfdrtu.htm>
-----
mode
You can use 10 characters. Each character has a specific meaning.
The first character Meaning
d The entry is a directory.
...
- The entry is an ordinary file.

The next nine characters divide into three sets of three characters each. The three characters in each set indicate, respectively, read, write, and execute permission of the file.

links
The number of links to the object. The minimum number of characters is 3. The maximum number of characters is 5. The characters are right justified, and they include blanks.

owner
The owner of the object. The minimum number of characters is 8. The maximum number of characters is 10. The characters are left justified, and they include blanks. This field contains the user profile name of the object owner. However, for anonymous FTP sessions, this field contains the owner ID number.

group
The owner of the object. The minimum number of characters is 8. The maximum number of characters is 10. The characters are left justified, and they include blanks. This field contains the user profile name of the group. However, if there is no group, this field contains the group ID number. The field also contains the group ID number for an anonymous FTP session.

size
The size of the object. The minimum number of characters is 7. The maximum number of characters is 10. The characters are right-aligned, and they include blanks. When there is no size for the object, the default is zero.

datetime
The 12 character modification time. The characters are left-aligned, and they include blanks. This is the format of this field when the modification time is within the previous 180 days:

Mmm dd hh:mm

This is the format of this field when the modification time is not within the previous 180 days:

Mmm dd yyyy

name
The variable length name of the object, which precedes a CRLF (carriage return, line feed pair). The name may include blanks.


Here is an example of the UNIX style format:

drwxrwxrwx 4 QSYS 0 51200 Feb 9 21:28 home


Consider this information as you work with UNIX format data that is returned by the LIST subcommand: When LISTFMT=1, the LIST content varies for QSYS.LIB files depending on the NAMEFMT setting:

When NAMEFMT=1, you will see only the QSYS.LIB file names.
When NAMEFMT=0, you will see both the QSYS.LIB file names and the names of the members in the file or files.

Parent topic: Configuring FTP servers for graphical FTP clients and Web tools
Related reference
File and directory entries in i5/OS format
SITE (Send Information Used by a Server System)
LIST (File List)
NLST (Name List)
-----
LorenAmelang
OutBack Whiz
 
Posts: 37
Joined: Mon Oct 26, 2009 10:51 pm
My RE system: (being majorly revised...)

Re: How to remote download the AXS port’s SD data (csv. file

Postby intellact on Fri Mar 04, 2016 3:54 pm

Thanks for your additional information, Loren Amelang.

The latest word I received from OutBack was a bit different, but may mean the same thing. (I confess, I'm not entirely clear on the full implications of your post.)

Here are the two bits of new I got today:

OutBack wrote:We are attempt to re-prioritize the threads to allow FTP to execute.

The NULs issue within CSV files has been fixed on AXS Port.

So hopefully, we'll have a solution available soon.
Andrew Welch, creator of WattPlot
User avatar
intellact
OutBack Czar
 
Posts: 602
Joined: Mon Nov 28, 2005 4:20 am
Location: Ontario, Canada

Re: How to remote download the AXS port’s SD data (csv. file

Postby LorenAmelang on Fri Mar 04, 2016 9:37 pm

Andrew,

They may well have some internal thread priorities wrong as well, but the first problem is the wrong number of characters in the permissions section at the beginning of each line of the returned directories. It needs to look exactly like a proper Linux directory, but it is one character short. If you open the "log drawer" in Cyberduck, you can see all of your directories go past with all the proper files showing and no delays or drops. But each line is wrong, so a proper FTP client ignores them all.

I do find WS_FTP95, which ignores the format error, loses the connection after eight or ten files get through. Maybe that is the thread priority issue.

I received this reply to today's inquiry about the update promised for last month:

Apparently 3.13 will be skipped and 3.15 will be out in a couple of weeks. Hopefully it will be worth the wait. Meanwhile, the backend server did get some updates and bug fixes which could help solve some of your issues.

After reading your quote, I'm wondering if they outsource this development to some far away country - that would help explain why the local support people seem so disconnected from the process.
LorenAmelang
OutBack Whiz
 
Posts: 37
Joined: Mon Oct 26, 2009 10:51 pm
My RE system: (being majorly revised...)

Re: How to remote download the AXS port’s SD data (csv. file

Postby intellact on Thu Apr 20, 2017 5:25 pm

Perhaps it is worth updating this thread...

As of firmware 003.015.001, it appears that we now have full FTP access to the files on the AXS Port's SD Card. =D> I haven't tried to access the MATE3 under the 003.015.001 release, but will try that soon.

Note that the directory holding the hourly data files on the SD Card is called "DATALOGS", not "LOGFILES" as documented in the AXS Port User's Guide. :roll:

We already have one WattPlot tool that now downloads these files and summarizes them automatically. :smile: Perhaps I'll get around to adding that feature to WattPlot SumMATE as well!
Andrew Welch, creator of WattPlot
User avatar
intellact
OutBack Czar
 
Posts: 602
Joined: Mon Nov 28, 2005 4:20 am
Location: Ontario, Canada

Re: How to remote download the AXS port’s SD data (csv. file

Postby intellact on Mon Apr 24, 2017 7:31 am

Earlier, I wrote:We already have one WattPlot tool that now downloads these files and summarizes them automatically. :smile: Perhaps I'll get around to adding that feature to WattPlot SumMATE as well!

Since a WattPlot client asked for clarification on this, I though others might need it as well.

WattPlot SumMATE processes the raw data found on a MATE3 or AXS Port SD card and summarizes it into useful information that can be graphed (by month), plotted or summarized (by day), converted to summary spreadsheets, analysed to the second, and more. :grin: It can also copy or move the data log files, backing up data and creating space on the SD card.

The catch is that at the present time, because of the issues with the earlier MATE3/AXS Port firmware releases, you need to REMOVE the SD card from the device and make it readable by the PC running the WattPlot SumMATE software.

Our new tool WattPlot ACKcess (now going into field trials), takes advantage of the fact that OutBack has fixed the FTP issues, so it can go directly to the SD Card while it is still in the MATE3 or AXS Port, download the files, and do the same great functions! \:D/ This is a huge step forward for total remote system control and information gathering.

Imagine - one tool that allows you to remotely...

    See ALL of your system data, in real time (or past history), whenever you choose.
    Change ANY system setting or send a command to execute ANY system function.
    Gather past data logs, automatically, whenever a connection is established, and turn them into real information.
    Program your system beyond the functionality built-in, defining multi-condition triggers and responses (including email notifications) so that it will have the functionality that YOU need.

Exciting times... :cool:
Andrew Welch, creator of WattPlot
User avatar
intellact
OutBack Czar
 
Posts: 602
Joined: Mon Nov 28, 2005 4:20 am
Location: Ontario, Canada


Return to AXS Port

Who is online

Users browsing this forum: No registered users and 2 guests