suggestion: more settings!

[BLovegrove]

Hi there! Love your work, found this project when looking for a version of tidal-dl that was a little more recently updated. Made a fork to test out some very hacky proof-of-concept ideas and thought I'd mention them in here! Would like to preface this by saying I'm super new to the nitty-gritty workings of docker containers and the level of skill I'm seeing in your code is well above me so take anything I say with a huge grain of salt (':

For starters, I store all my media on an Unraid server so I was looking for something with accessible in some way via the Unraid UI for convenient access to my media folders, and given how Unraid manages docker containers booting the docker up once per music download seemed like a huuuuuge time sink so the first thing I did was replace the entry-point and default CMD with a replacement 'sleep infinity' to make the docker persistent making repeat download commands much easier. Is there some way something similar could be added in a slightly less jank way? could this be done with an ENV when you're setting up the container?

The other thing I wanted to suggest was adding some environment variables to change up certain things like how a playlist is downloaded, whether or not to include extra metadata like cover images, artist bios, etc, and some way to alter how files are saved as not everybody's naming conventions will be compatible and being able to adjust it to fit your existing library would be a godsend.

Also it seems like the android token for HiRes downloads is being processed before its stored in the .token file, would it be doable to have it stored raw so you can just go in and copy/paste the new token in if it ever stops working? (or did I misunderstand how the token time-out thing works?)

Thanks for sharing all this! Its nice to see alternatives given how long tidal-dl has been basically the only option for keeping a local copy of your tidal library (:

[ebb-earl-co]

Thank you for having opened a conversation, @BLovegrove! I'll try to address your points in bullets here, but first I'd like to say that if the nitty-gritty of Docker or how I've done Python is not clear, then that's something for me to work on!

• Interesting question about the Unraid environment. I've not used Unraid and I barely know anything about it, but the gist from your comment that I'm getting is that you'd like to use tidal-wave via Docker to repeatedly execute downloads? If that's the case, then the Dockerfile could simply be changed to have a /bin/bash entry point or similar, and then the docker run command would just start with python3 -m tidal-wave ... the rest. I've been trying to make tidal-wave into a "one-click" utility that attempts to download the desired data and exits cleanly, but I suppose there's nothing wrong with a container that is long-lived, and is really just a wrapper around the Python, FFmpeg, and tidal-wave requirements... Let me fiddle with a few things and respond to this once I've got a more long-lived Dockerfile option (maybe even an ENV, as you point out 🤔 )
• How the playlist is downloaded, extra metadata flag, whether to grab the artist bio, etc. is something that I thought about a decent amount when I first forked Tidal-Media-Downloader, so I'll give you my reasoning here:
  In my mind, access to the TIDAL API is the limiting factor of tidal-wave, or any other media downloader application. So, the program should be greedy and retrieve everything that it can while it has access. Moreover, I think of a track or an album or a playlist as one entity, and all the metadata are not accessories, they're parts of the "thing" ™️ that is trying to be retrieved. For this reason, I am loath to separate the different stages. That being said, I wrap some of the "less important" metadata in lazy try/except blocks because they're not utterly necessary to the data desired, such as an artist's bio JSON or album review JSON, so I understand that there is a hierarchy of information desired, obviously with the bytes of the audio itself at the top.
  Separately, I know the feeling of having some software that you wish just had this toggle or that toggle, because post-processing the thing it outputs is annoying or inefficient. However, when looking at the config complications of the original Tidal-Media-Downloader, I was put off from developing so many toggles. It's much better, in my opinion, to have a utility that tries to grab everything that it can, reproduced as faithfully as it can, and allow for infinite post-processing.
• Can you elaborate on the point of how playlists are downloaded? I am not completely satisfied with playlist retrieval at the moment, so I am all ears on this topic.
• On the point of the Android token, the fundamental issue is that these tokens last for 24 hours, that's it. There isn't a refresh possible from code unless you go back into your Android device and start playing new tracks or videos. This is fundamental to the TIDAL security model on the mobile devices, I'm pretty sure. However, yes, there is a simple way of editing the value of the token if that's all that you want to do: the android-tidal.token file is simply Base64-encoded JSON, so you can use any tool you like to decode the base64 (it's not encrypted, just lightly obfuscated using Python's base64 library) and edit the JSON with a different access token value. E.g. with Bash on Ubuntu: $ cat ~/.config/tidal-wave/android-tidal.token | base64 -d will show the JSON that you can then copy/paste into an editor, change the value for "access_token" and re-encode with base64, saved right back as ~/.config/tidal-wave/android-tidal.token. I advise, if you do that, to make the "session_id" value to an empty string, though, as the session almost certainly will have changed if your access token has expired. To finish the process here, what you'd do with the JSON you're editing (saved as, say, ~/Desktop/tidal.json for example) is $ cat ~/Desktop/tidal.json | base64 -w 0 > ~/.config/tidal-wave/android-tidal.token

Keep the questions and requests coming!

[BLovegrove]

I'll copy your format here and reply in bullet-points - thanks for your speedy and detailed response!!

• Awesome! Yeah, Unraid displays dockers like the image below. you add your container by filling out all your ENVs, volume mappings, etc, and it boots up into this list where you can view stdout logs, directly access the console, etc. - ideally I'd like to be able to hit the console button and start downloading songs via the command-line or even use docker exec to script it in some capacity. To do this, the container needs to be long-lived. Hope that clears things up a bit! I'd highly recommend looking into Unraid if you run the kind of software in my screenshot, its a very neat OS!
  
• I totally understand how daunting all the toggles would be, I was more thinking a single toggle to download extra files, or not. when enabled, you get all the bonus metadata and can do with that collection whatever you like, but for my case and I think a lot of people who will be using this kind of software, I have another back-end (beets) that will be managing my library. fixing metadata, fingerprinting, adding cover art as a de-interlaced jpeg for my ipod to handle, that kind of thing. so the extra meta-data is just extra files I have to run through and clean up manually later and presumably adds more time to the download process (every ms helps when you're downloading like 5000 songs lol). Would something like that be doable? Even a flag like the '--audio-format' flag to '--disable-extras' would be perfect.
• re: playlist - tidal-dl has an option to disable the playlist folder structure, meaning instead of putting files in 'Playlist/PlaylistName/ArtistName/etc' it saves them as if you had queued up each album individually i.e: 'ArtistName/AlbumName/etc' again without all the extra data that comes along with playlists - this goes back to my previous example about having a separate back-end for managing my metadata as grouping it like it is now can confuse the importer. You can also get duplicate songs that way which could confuse media front-ends etc. A single flag or ENV toggle for this would also be perfect!
• thanks for that explanation! That makes more sense now, I wonder if it would be doable to have the program grab your token automatically if running straight on windows rather than via docker? a windows token works too right? (just based on the input when it asks you to enter your platform android or windows).

A neat add-on for the android token part that I just thought of - if your docker is persistent like I mentioned in point 1, you could store the date your android token was active and check that stored date when someone tries to download a track. if 24hrs has passed, re-prompt. probably a lot of work for relatively little convenience but just thought I'd throw it out there in case you're interested (:

[ebb-earl-co]

@BLovegrove

• when running Docker containers on Unraid, is there an option to customize the CMD that is run with the container? If so, then I've got a very simple change to Dockerfile that would allow you to have a long-lived container while the default behavior is to invoke tidal-wave. Docker's ENTRYPOINT and CMD are neat...
• In your opinion, what are the extra files? From my perspective, adding the metadata to the audio file is a necessary step: everything else is up for classification, even the cover image and its addition to the file (even though I'm not over the moon about leaving that step out)
• In terms of the playlists, what you're describing is not writing to a Playlist subdirectory at all? Instead, the TIDAL playlist would be an instruction for downloading many albums, is that right?
• The Windows authentication token flow is not the ideal way of getting the access token because it requires setting up a man-in-the-middle attack on oneself in order to intercept HTTPs traffic and glean it from there. Furthermore, there have been a few issues filed in which the access token acquired from Windows retrieves encrypted audio bytes; I have no plans to figure out how to work around that issue.
• The automatic fetching of tokens for HiRes and Sony 360 content isn't where I want to direct my energy for now. This is because TIDAL's security model is explicitly to get users to re-interact with their devices every 24 hours exactly so that credentials can't be automated. The only reason that the default path for tidal-wave is to use a spoofed Fire TV is so that it behaves like Tidal-Media-Downloader; i.e. the credentials were leaked years ago, and TIDAL hasn't patched them, so I don't feel much culpability, it's just that Fire TVs and the like authenticate with the TIDAL API differently and it causes confusion with how to interact with tidal-wave...

[BLovegrove]

• I'm not 100% certain, these are the options available when setting up a container - any of this look right? I'm hunting down some more specific documentation on how Unraid runs its containers, will update this when I find something!
  
• I should clarify, all the embedded meta-data is great! By extra files I'm referring to anything that isn't the .flacs (or other music files) that are downloaded. i.e, the artist image, the stand-alone album art, .json files, etc. as in my experience none of the music players or library front-ends can even read that data. It's a nice option for those who want to archive it, but for the music listening experience that extra info is usually grabbed from sources like last fm, musicbrainz, etc and just takes up space and processing time.
• Yes! mostly... Rather than downloading all the albums in the playlist, it's just downloading the individual tracks as it already does, but instead of placing them in the playlist subdirectories, it attempts to place them in the main 'Artist > Album (Year) > Tracks' structure. This way if you have multiple playlists with the same tracks, you don't end up with duplicates. It also makes managing it with a separate back end easier. Think of it like scrolling through your playlist, copy-pasting single track links one at a time but you can do it for 1000+ songs at a time
• Darn, sounds like a mess as most things usually are in windows. Oh well, thanks for the explanation!
• Fair enough, didn't realize it worked that way - guess I'll keep an eye out and see if I can't implement it myself if any of that changes in the future (:

[ebb-earl-co]

@BLovegrove I've just pushed a new commit to the main branch, trunk, so can you try out the new image? https://github.com/ebb-earl-co/tidal-wave/pkgs/container/tidal-wave/173934730?tag=trunk
In particular, where there's the option in Unraid to pass a custom command for the Docker container, you only need to specify --entrypoint="/bin/bash", and that the image is from the trunk branch, rather than the default, latest. I don't know exactly how to do these things with Unraid, so give it a go and let me know if it works.

[BLovegrove]

darn, fails to stay open and the log is empty

[BLovegrove]

oh! @ebb-earl-co scratch that, you can simply add --interactive --tty to the extra parameters as well and it'll stay open

[ebb-earl-co]

@BLovegrove how is the experience with Unraid? Is it working the way you expected?

[BLovegrove]

Yeah, sorry! I got busy. It's behaving as expected with the extra 'extra parameters' I mentioned, might create a new fork now to demonstrate the extra settings that I'm talking about - is it alright if I ping you with the occasional question here? Or would you rather a new thread/not at all

[ebb-earl-co]

If you've got a feature request, go ahead and open an Issue!

[ebb-earl-co]

@BLovegrove in the trunk branch, I've just merged a feature that you had asked about: a new flag, --no-flatten, that will download a playlist's or mix's tracks/videos without reordering them and placing them in either Playlists or Mixes subdirectory. Try it out!

[BLovegrove]

Awesome thank you! I'll have a look when my ipod battery replacement finally arrives and I can get back to curating my music collection lol