Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable 128K virtual memory via external SPI SRAM #6994

Merged
merged 15 commits into from
Mar 15, 2021

Commits on Jan 19, 2020

  1. Enable virtual memory 128K+ via external SPI SRAM

    Provides a transparently accessible additional block of RAM of 128K to
    8MB by using an external SPI SRAM.  This memory is managed using the UMM
    memory manager and can be used by the core as if it were internal RAM
    (albiet much slower to read or write).
    
    The use case would be for things which are quite large but not
    particularly frequently used or compute intensive.  For example, the SSL
    buffers of 16K++ are a good fit for this, as are the contents of Strings
    (both to avoid main heap fragmentation as well as allowing Strings of
    >30KB).
    
    A fully associative LRU cache is used to limit the SPI bus bottleneck,
    and background writeback is supported.
    
    Requires `ESP.enableVM()` call to actually add the VM subsystem.  If
    this routine is not called, then the entire VM routines should not be
    linked in to user apps, so there should be no space penalty w/o it.
    
    UMM `malloc` and `new` are modified to support internal and external
    heap regions.  By default, everything comes from the standard heap, but
    a call to `ESP.setExternalHeap()` before the allocation (followed by a
    call to `ESP.resetHeap()` will make the allocation come from external
    RAM.  See the `virtualmem.ino` example for use.
    
    If there is no external RAM installed, the `setExternalHeap` call is a
    no-op.
    
    The String and BearSSL libraries have been modified to use this external
    RAM automatically.
    
    Theory of Operation:
    
    The Xtensa core generates a hardware exception (unrelated to C++
    exceptions) when an address that's defined as invalid for load or store.
     The XTOS ROM routines capture the machine state and call a standard C
    exception handler routine (or the default one which resets the system).
    
    We hook into this exception callback and decode the EXCVADDR (the
    address being accessed) and use the exception PC to read out the
    faulting instruction. We decode that instruction and simulate it's
    behavior (i.e. either loading or storing some data to a
    register/external memory) and then return to the calling application.
    
    We use the hardware SPI interface to talk to an external SRAM/PSRAM,
    and implement a simple cache to minimize the amount of times we need
    to go out over the (slow) SPI bus. The SPI is set up in a DIO mode
    which uses no more pins than normal SPI, but provides for ~2X faster
    transfers.  SIO mode is also supported.
    
    NOTE: This works fine for processor accesses, but cannot be used by
    any of the peripherals' DMA. For that, we'd need a real MMU.
    
    Hardware Configuration (only use 3.3V compatible SRAMs!):
    
      SPI byte-addressible SRAM/PSRAM: 23LC1024 or smaller
        CS   -> GPIO15
        SCK  -> GPIO14
        MOSI -> GPIO13
        MISO -> GPIO12
     (note these are GPIO numbers, not the Arduino Dxx pin names.  Refer
      to your ESP8266 board schematic for the mapping of GPIO to pin.)
    
    Higher density PSRAM (ESP-PSRAM64H/etc.) should work as well, but
    I'm still waiting on my chips so haven't done any testing.  Biggest
    concern is their command set and functionality in DIO mode.  If DIO
    mode isn't supported, then a fallback to SIO is possible.
    
    This PR originated with code from @pvvx's esp8266web server at
    https://github.com/pvvx/esp8266web (licensed in the public domain)
    but doesn't resemble it much any more.  Thanks, @pvvx!
    
    Keep a list of the last 8 lines in RAM (~.5KB of RAM) and use that to
    speed up things like memcpys and other operations where the source and
    destination addresses are inside VM RAM.
    
    A custom set of SPI routines is used in the VM system for speed and code
    size (and because the core cannot be dependent on a library).
    earlephilhower committed Jan 19, 2020
    Configuration menu
    Copy the full SHA
    8788c76 View commit details
    Browse the repository at this point in the history

Commits on Dec 22, 2020

  1. Configuration menu
    Copy the full SHA
    f3927f1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e333d50 View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2021

  1. Configuration menu
    Copy the full SHA
    e1d580e View commit details
    Browse the repository at this point in the history
  2. Tested and running with latest UMM IRAM heap changes

    Ran sanity tests successfully on 23LC1024 128K SRAM.  Adjusted defines
    to ensure UMM knows about our heap.
    earlephilhower committed Jan 9, 2021
    1 Configuration menu
    Copy the full SHA
    5bb31a0 View commit details
    Browse the repository at this point in the history
  3. Unify exception handler logic between non32xfer/vm

    Use same ASM macro from @mhightower83's non32xfer handler in the VM
    handler, avoid potential GCC optimization issues.
    earlephilhower committed Jan 9, 2021
    Configuration menu
    Copy the full SHA
    7619858 View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2021

  1. Configuration menu
    Copy the full SHA
    8474edf View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2021

  1. Configuration menu
    Copy the full SHA
    1e1bc49 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a129c01 View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2021

  1. Configuration menu
    Copy the full SHA
    8d99fd9 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2021

  1. Configuration menu
    Copy the full SHA
    d106e88 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2021

  1. Adjust UMM to only map 256K of 1M PSRAM

    Because UMM manages RAM in 8 byte chunks, attempting to manage the
    entire 1M available space on a 1M PSRAM causes the block IDs to
    overflow, crashing things at some point.  Limit the UMM allocation to
    only 256K in this case.  The remaining space can manually be assigned to
    buffers/etc. managed by the application, not malloc()/free().
    earlephilhower committed Jan 25, 2021
    Configuration menu
    Copy the full SHA
    fc85d18 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2021

  1. Configuration menu
    Copy the full SHA
    3fa9779 View commit details
    Browse the repository at this point in the history
  2. Update boards.txt

    earlephilhower committed Mar 14, 2021
    Configuration menu
    Copy the full SHA
    bd302ca View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2021

  1. Configuration menu
    Copy the full SHA
    c433fa8 View commit details
    Browse the repository at this point in the history