SD file system, request to add file.CreationDate (). #6992

NicoZuid · 2020-01-06T17:22:11Z

In version 2.5.0 was a directory of utilities included.
A few years ago I found an example on the internet that made it possible to make the creation date of an SD file visible.
In version 2.6.3, Arduino SdFat Library by William Greiman is still used. (2009)

..\2.6.3\librarie\ESP8266\SdFat\src\Fatlib\FatFile.cpp

This file contains code to process the SD file creation date / time.
Is it possible to add this to the options of the SDFS library?

// set up variables using the SD utility library functions:
Sd2Card card;
SdVolume volume;
SdFile root;
File sdFile;

SdFile::dateTimeCallback(dateTime);                  // set callback for SD file date


//=============================================================================
// this function is called by SD if it wants to know the time
void dateTime(uint16_t* date, uint16_t* time)
{
  // return date using FAT_DATE macro to format fields
  *date = FAT_DATE(year(), month(), day());
  // return time using FAT_TIME macro to format fields
  *time = FAT_TIME(hour(), minute(), second());
}
//=============================================================================

void handleSDfilelist() {
  const char* br = "<br>\n";
  message = "<!DOCTYPE html> <html> <head> <title>SD files</title></head> <body>";
  message += "<table style='font-size:14px'><tr><td><b>SD files</b></td></tr>";

root.openRoot(volume);
  dir_t p;
  root.rewind();
  delay(1);  //yield();   //201909 probleem WDT restart oplossen (?)
  while (root.readDir(p) > 0) {
    if (p.name[0] == DIR_NAME_FREE) break;
    // skip deleted entry and entries for . and  ..
    if (p.name[0] == DIR_NAME_DELETED || p.name[0] == '.') continue;
    // only list subdirectories and files
    if (!DIR_IS_FILE_OR_SUBDIR(&p) ) continue;
    message += "<tr><td><a href=\"../";
    for (uint8_t i = 0; i < 11; i++) {
      if (p.name[i] == ' ') continue;
      if (i == 8) {
        message += ".";
      }
      message +=  (char)p.name[i];
    }
    message += "\">";

    // print file name with possible blank fill
    for (uint8_t i = 0; i < 11; i++) {
      if (p.name[i] == ' ') continue;
      if (i == 8) {
        message += ".";
      }
      message += (char)p.name[i] ;
    }
    message += "</a></td><td>";
    message += p.fileSize;
    message += F("</td>");
    SDdatum(p.creationDate, p.creationTime);
    message += ioBuffer;
    message += "   ";
    SDdatum(p.lastWriteDate, p.lastWriteTime);
    message += ioBuffer;
  }
  message += "</tr> \n";

  message += "</table></body></html> \n";
  server.send(200, "text/html", message);
}

earlephilhower · 2020-01-06T18:10:38Z

I tried to mirror the ESP32's File interface when I rejiggered things. Do they already have a call like that? It's almost trivial to add (requires a minor change to mklittlefs to add a new attribute, but it's binary backwards compatible w/existing FSes), but I want to keep things as closely tied as possible.

One option in the meantime is to use the SdFat library directly and bypass the Arduino core File stuff altogether. I haven't tried it, but no changes were made to the SDFat lib from Greiman except for compatibility changes to the low-level interfaces in my fork.

NicoZuid · 2020-01-06T20:50:35Z

My knowledge of the C ++ programming language is too limited to come up with a solution myself.

I have now added the code from my program which shows the date.
Unfortunately I am unable to get version 2.5.0 up and running again at the moment.
It seems that the IDE version change to 1.8.10 makes it impossible to go back to the old version.

NicoZuid · 2020-01-07T06:22:53Z

Just probably found the solution in the sdFat examples of the ESP8266SdFat library.
In the example "Timestamp" the following code is used:
f.printFatTime (d.creationDate); / f.printFatTime (d.creationTime);
and also
f.printFatDate (d.lastAccessDate);
I can continue again.
mvg Nico

earlephilhower · 2020-01-07T17:06:49Z

Yes, the actual addition isn't a big deal. We already have enough info to get it with the existing code. I checked the ESP32 and they don't have such a call, but that's not to say we can't. Let me see if I can throw something together tonight.

For SDFS and LittleFS, enable a creation time accessor for files and Dir iterators, similar to the existing fileTime/getLastWrite calls. Remove spurious Dir::getLastWrite method (the proper and only documented way is really Dir::fileTime). Update json to point to new mklittlefs which copies the creation date of files to the image. Fixes esp8266#6992

earlephilhower · 2020-01-08T13:41:33Z

@NicoZuid, can you give PR #7000 (!) a try on your SD setup and report back? Mine is presently buried somewhere on my desk so all I was able to verify personally was the LittleFS changes.

NicoZuid · 2020-01-08T17:36:58Z

I'm going to see if I succeed.

NicoZuid · 2020-01-08T17:55:21Z

Unfortunately this would be my first time that I would have to replace a cpp file in a library with a test version.
As I have done now I get compile errors.
Is there somewhere described the correct way to include the test version of SDFS.cpp in my project?

earlephilhower · 2020-01-08T17:56:50Z

https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/checking-out-pull-requests-locally

NicoZuid · 2020-01-08T18:16:29Z

Instruction is not yet in line with what I see on my screen.
Do I have sufficient rights to do this?

NicoZuid · 2020-01-09T17:27:47Z

I manually transferred the source codes of 7 files to my PC via PR # 7000.
I get the (old) creationDateTime printed correctly (SDFS adapted example LittleFS)
Unfortunately not the changes to PR #R6985.
So no NEW dates (from today) are written on the SD.

I have looked at the source code of SDFS.h and it includes the change from PR #6985. Unfortunately it doesn't work.

Debug Messages



Connecting to groen4ZUID
...............WiFi connected
IP address: 
192.168.2.4
Contacting Time Server

Now is : 2020-01-09 20:06:43

Mount SDFS
Listing directory: /
  FILE: System Volume Information  SIZE: 0    CREATION: 2020-01-03 18:44:58
  LAST WRITE: 2020-01-03 18:45:00
  FILE: hello.txt  SIZE: 61    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: temptab.txt  SIZE: 317    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: styl1.css  SIZE: 1023    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: FSBrowser.ino  SIZE: 8564    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: Windows.txt  SIZE: 13    CREATION: 2020-01-04 06:29:18
  LAST WRITE: 2020-01-04 06:29:58
  FILE: temp_grafiek.html  SIZE: 7218    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: example.txt  SIZE: 0    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: datalog.txt  SIZE: 22180    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: default.txt  SIZE: 0    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: callback.txt  SIZE: 1    CREATION: 2014-10-01 20:30:40
  LAST WRITE: 2014-10-02 20:30:42
  FILE: stamp.txt  SIZE: 0    CREATION: 2014-11-10 01:02:02
  LAST WRITE: 2014-11-11 04:05:06
Deleting file: /hello.txt
File deleted
Writing file: /hello.txt
File written
Appending to file: /hello.txt
Message appended
Listing directory: /
  FILE: System Volume Information  SIZE: 0    CREATION: 2020-01-03 18:44:58
  LAST WRITE: 2020-01-03 18:45:00
  FILE: hello.txt  SIZE: 13    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: temptab.txt  SIZE: 317    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: styl1.css  SIZE: 1023    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: FSBrowser.ino  SIZE: 8564    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: Windows.txt  SIZE: 13    CREATION: 2020-01-04 06:29:18
  LAST WRITE: 2020-01-04 06:29:58
  FILE: temp_grafiek.html  SIZE: 7218    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: example.txt  SIZE: 0    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: datalog.txt  SIZE: 22180    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: default.txt  SIZE: 0    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: callback.txt  SIZE: 1    CREATION: 2014-10-01 20:30:40
  LAST WRITE: 2014-10-02 20:30:42
  FILE: stamp.txt  SIZE: 0    CREATION: 2014-11-10 01:02:02
  LAST WRITE: 2014-11-11 04:05:06
The timestamp should be valid above
Now unmount and remount and perform the same operation.
Timestamp should be valid, data should be good.
Now mount it
Reading file: /hello.txt
Read from file: Hello World!
Listing directory: /
  FILE: System Volume Information  SIZE: 0    CREATION: 2020-01-03 18:44:58
  LAST WRITE: 2020-01-03 18:45:00
  FILE: hello.txt  SIZE: 13    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: temptab.txt  SIZE: 317    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: styl1.css  SIZE: 1023    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: FSBrowser.ino  SIZE: 8564    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: Windows.txt  SIZE: 13    CREATION: 2020-01-04 06:29:18
  LAST WRITE: 2020-01-04 06:29:58
  FILE: temp_grafiek.html  SIZE: 7218    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: example.txt  SIZE: 0    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: datalog.txt  SIZE: 22180    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: default.txt  SIZE: 0    CREATION: 2000-01-01 01:00:00
  LAST WRITE: 2000-01-01 01:00:00
  FILE: callback.txt  SIZE: 1    CREATION: 2014-10-01 20:30:40
  LAST WRITE: 2014-10-02 20:30:42
  FILE: stamp.txt  SIZE: 0    CREATION: 2014-11-10 01:02:02
  LAST WRITE: 2014-11-11 04:05:06

earlephilhower · 2020-01-11T17:15:59Z

I tried on my own system and SD create/write stamps are looking good and matching what my Linux box reports for the same SD card, so I think we're good.

NicoZuid · 2020-01-11T18:28:29Z

Do you mean that the dates written on the sd via your Linux computer are also visible via the arduino?

In my opinion, the problem is not to show the written date stamps.
New creation and/or write date stamps via SDFS are not correct.

WRONG:
FILE: hello.txt SIZE: 13 CREATION: 2000-01-01 1:00:00
LAST WRITE: 2000-01-01 1:00:00
CORRECT:
FILE: Windows.txt SIZE: 13 CREATION: 2020-01-04 06:29:18
LAST WRITE: 2020-01-04 06:29:58

(LittleFS with the same program gives correct date/time stamps)

earlephilhower · 2020-01-12T00:41:25Z

I took my 128MB card with old photos and ran the modified listfiles example. Compared the create/write times to the ones returned from stat on my machine and they matched.

NicoZuid · 2020-01-12T05:58:27Z

Sorry but I think a modified list files example does NOT change a creation date or lastWrite date.

I used the example LittleFS_Timestamp and changed all code LittleFS to SDFS.
This example creates and changes the hello.txt file.
Date and time stamps of hello,txt remain unchanged 2000-01-01 01:00:00

Don't forget to remove SDFS.format ();

earlephilhower · 2020-01-19T18:36:38Z

@NicoZuid , I just pushed an update to #7000 which I was able to test. I was setting a callback for date/time, but never telling the SdFat library to use it, so create/write times never got updated in the device.

Just make sure that NTP is configured and running before you do file ops (same as other filesystems, of course!) and then you should get the proper dates on all files made on the device, now.

Example file created/written on the 8266:

newfil4.txt		14	CREATION: 2020-01-19 18:31:00	LAST WRITE: 2020-01-19 18:31:14

NicoZuid · 2020-01-20T06:46:44Z

@earlephilhower , This morning (CET 7:25 AM) I tested the new version, I can confirm that it now works correctly.

_Listing directory: /
  FILE: hello.txt  SIZE: 13    CREATION: 2020-01-20 07:27:24
  LAST WRITE: 2020-01-20 07:27:28_

I always use TimeLib.h in my programs (instead of time.h)
TimeLib.h makes it easier, for example, to check whether NTP has started correctly.

while (year(time(NULL)) < 2020) {
    delay(500);
    Serial.print(".");
  }

* Add fileCreation/getCreation create-time accessors For SDFS and LittleFS, enable a creation time accessor for files and Dir iterators, similar to the existing fileTime/getLastWrite calls. Remove spurious Dir::getLastWrite method (the proper and only documented way is really Dir::fileTime). Update json to point to new mklittlefs which copies the creation date of files to the image. Fixes #6992 * Remove malloc(), use stack vars for temp names LFS filenames are limited in size and generally very small. Use a stack variable instead of a dynamic allocation when performing full-path computations. * Replace "Creation" w/"CreationTime" in FS accessor Per review, `getCreation` -> `getCreationTime`, `fileCreation` -> `fileCreationTime`. The names `fileTime()` and `getLastWrite()` are inherited from ESP32 implementation and unchanged. * Add creation time to listfiles SD example * Enable SdFat's sateTime callback for timestamping SdFat requries the dateTimeCallback call (global for everything) to update dates and times on created files. Because the callback signature doesn't have space for us to provide any parameters, we cannot get the the File, Dir, or FS object's dateTimeCB member. Instead, just go with `time(null)` as the callback function which is right in all but the most esoteric cases. * Correct DOS year/month offset in dateTime callback * Fix docs to match new xxxCreationTime() API names Co-authored-by: Develo <[email protected]>

earlephilhower mentioned this issue Jan 8, 2020

Add fileCreation/getCreation create-time accessors #7000

Merged

earlephilhower added the waiting for feedback Waiting on additional info. If it's not received, the issue may be closed. label Jan 19, 2020

earlephilhower closed this as completed in #7000 Feb 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SD file system, request to add file.CreationDate (). #6992

SD file system, request to add file.CreationDate (). #6992

NicoZuid commented Jan 6, 2020 •

edited

Loading

earlephilhower commented Jan 6, 2020

NicoZuid commented Jan 6, 2020

NicoZuid commented Jan 7, 2020

earlephilhower commented Jan 7, 2020

earlephilhower commented Jan 8, 2020

NicoZuid commented Jan 8, 2020

NicoZuid commented Jan 8, 2020

earlephilhower commented Jan 8, 2020

NicoZuid commented Jan 8, 2020

NicoZuid commented Jan 9, 2020 •

edited

Loading

earlephilhower commented Jan 11, 2020

NicoZuid commented Jan 11, 2020

earlephilhower commented Jan 12, 2020

NicoZuid commented Jan 12, 2020

earlephilhower commented Jan 19, 2020 •

edited

Loading

NicoZuid commented Jan 20, 2020

SD file system, request to add file.CreationDate (). #6992

SD file system, request to add file.CreationDate (). #6992

Comments

NicoZuid commented Jan 6, 2020 • edited Loading

earlephilhower commented Jan 6, 2020

NicoZuid commented Jan 6, 2020

NicoZuid commented Jan 7, 2020

earlephilhower commented Jan 7, 2020

earlephilhower commented Jan 8, 2020

NicoZuid commented Jan 8, 2020

NicoZuid commented Jan 8, 2020

earlephilhower commented Jan 8, 2020

NicoZuid commented Jan 8, 2020

NicoZuid commented Jan 9, 2020 • edited Loading

Debug Messages

earlephilhower commented Jan 11, 2020

NicoZuid commented Jan 11, 2020

earlephilhower commented Jan 12, 2020

NicoZuid commented Jan 12, 2020

earlephilhower commented Jan 19, 2020 • edited Loading

NicoZuid commented Jan 20, 2020

NicoZuid commented Jan 6, 2020 •

edited

Loading

NicoZuid commented Jan 9, 2020 •

edited

Loading

earlephilhower commented Jan 19, 2020 •

edited

Loading