Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make sure tests do not leave any temp files behind #5345

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Make sure tests do not leave any temp files behind #5345

wants to merge 10 commits into from

Conversation

snejus
Copy link
Member

@snejus snejus commented Jul 2, 2024

Fixes #5229 and relates to #5285.

I have to admit thsi was a fairly tough task - I initially assumed that the problem lies
with how the tests are setup, and that we're probably missing some teardown_beets calls
here and there.

Unfortunately, it was not so simple. I came across several issues that gave rise to
leftover temporary files:

  1. fetchart, artresizer and play handling of temporary files. These plugins created
    isolated temporary files outside of the directories that tests clean up. You will find
    I added a couple of functions (namely get_module_tempdir) that force these plugins to
    create files in directories determined by their module names. This way we can clean up
    after them using the new CleanupModulesMixin.

  2. Tests that ran temporary directories setup twice, running _common.TestCase.setUp and
    test.helper.TestHelper.setup_beets. Both of these ran self.temp_dir = mkdtemp(),
    therefore the directories created by the initial setup persisted since those have been
    overridden and thus unreachable in the teardown. Here, I removed the setUp calls, see

    • test/plugins/test_embedart.py
    • test/test_importer.py
    • and test/test_plugins.py where setup_beets was called twice

  3. test/test_config_command.py attempted to manage the temporary directory by itself,
    where I found that tearDown failed to remove the directory for four tests. Could not
    figure out the cause, and found that delegating this task to TestHelper fixed the
    issue.

  4. Mediafile fixture removal depended on calling remove_mediafile_fixtures method, which
    test/plugins/test_zero.py failed to do. I made the fixtures to be created within the
    same temp_dir directory that gets removed in the teardown, so now they are taken care
    of automatically.

In summary, see the test modules that left files behind:

Temp files created by test/__init__.py
Temp files created by test/plugins/__init__.py
Temp files created by test/plugins/lyrics_download_samples.py
Temp files created by test/plugins/test_acousticbrainz.py
Temp files created by test/plugins/test_advancedrewrite.py
Temp files created by test/plugins/test_albumtypes.py
Temp files created by test/plugins/test_art.py
	/tmp/tmp11nicahe.jpg
	/tmp/tmp1bjmodum.png
	/tmp/tmped7nhls4.jpg
	/tmp/tmpflnzr9wz.jpg
	/tmp/tmpjngkauqs.png
	/tmp/tmpkzy9mn6t.jpg
	/tmp/tmpph_wmuea.jpg
	/tmp/tmps6gk58i_.jpg
	/tmp/tmpz2eji_o4.jpg
Temp files created by test/plugins/test_aura.py
Temp files created by test/plugins/test_bareasc.py
	/tmp/tmphl3kzhug
	/tmp/tmpnh2q6v02
	/tmp/tmpppw5qrhz
Temp files created by test/plugins/test_beatport.py
Temp files created by test/plugins/test_bucket.py
Temp files created by test/plugins/test_convert.py
Temp files created by test/plugins/test_discogs.py
Temp files created by test/plugins/test_edit.py
Temp files created by test/plugins/test_embedart.py
	/tmp/tmp1ayvqzhx
	/tmp/tmp58k6mdfx.jpg
	/tmp/tmp64c2lqiv
	/tmp/tmp6nar4kr5
	/tmp/tmp6u0d5dex
	/tmp/tmpacoq7w_f
	/tmp/tmpajnr_sxr
	/tmp/tmpasj16beh
	/tmp/tmpboyaixb5
	/tmp/tmpcrmcyt5r
	/tmp/tmpdomje5g3
	/tmp/tmplu3o6t6g
	/tmp/tmpns_xvkns
	/tmp/tmpo87o1h6o.jpg
	/tmp/tmpqem39h_j
	/tmp/tmprlzm18pb
	/tmp/tmpt22v4u6x
	/tmp/tmptp3rxdgv
Temp files created by test/plugins/test_embyupdate.py
Temp files created by test/plugins/test_export.py
Temp files created by test/plugins/test_fetchart.py
Temp files created by test/plugins/test_filefilter.py
Temp files created by test/plugins/test_ftintitle.py
Temp files created by test/plugins/test_hook.py
Temp files created by test/plugins/test_ihate.py
Temp files created by test/plugins/test_importadded.py
Temp files created by test/plugins/test_importfeeds.py
Temp files created by test/plugins/test_info.py
Temp files created by test/plugins/test_ipfs.py
Temp files created by test/plugins/test_keyfinder.py
Temp files created by test/plugins/test_lastgenre.py
Temp files created by test/plugins/test_limit.py
Temp files created by test/plugins/test_lyrics.py
Temp files created by test/plugins/test_mbsubmit.py
Temp files created by test/plugins/test_mbsync.py
Temp files created by test/plugins/test_mpdstats.py
Temp files created by test/plugins/test_parentwork.py
Temp files created by test/plugins/test_permissions.py
Temp files created by test/plugins/test_player.py
Temp files created by test/plugins/test_playlist.py
Temp files created by test/plugins/test_play.py
	/tmp/tmp6ohknmve.m3u
	/tmp/tmp8rw2z_j4.m3u
	/tmp/tmp9vi27ypx.m3u
	/tmp/tmpa_s66jh8.m3u
	/tmp/tmpb7h3cn3n.m3u
	/tmp/tmpexbmqvry.m3u
	/tmp/tmpinbqrt80.m3u
	/tmp/tmpql02hax5.m3u
	/tmp/tmpvbdzprsf.m3u
	/tmp/tmpzipim36x.m3u
Temp files created by test/plugins/test_plexupdate.py
Temp files created by test/plugins/test_plugin_mediafield.py
Temp files created by test/plugins/test_random.py
Temp files created by test/plugins/test_replaygain.py
Temp files created by test/plugins/test_smartplaylist.py
Temp files created by test/plugins/test_spotify.py
Temp files created by test/plugins/test_subsonicupdate.py
Temp files created by test/plugins/test_the.py
Temp files created by test/plugins/test_thumbnails.py
Temp files created by test/plugins/test_types_plugin.py
Temp files created by test/plugins/test_web.py
Temp files created by test/plugins/test_zero.py
	/tmp/tmp3ub9xmzy
Temp files created by test/rsrc/beetsplug/test.py
Temp files created by test/rsrc/convert_stub.py
Temp files created by test/testall.py
Temp files created by test/test_art_resize.py
	/tmp/tmp3p7p60ih.jpg
	/tmp/tmp8exclgit.jpg
	/tmp/tmpkrrjsitl.jpg
	/tmp/tmpw6n8ee8e.jpg
	/tmp/tmpygws_0aw.jpg
Temp files created by test/test_autotag.py
Temp files created by test/test_config_command.py
	/tmp/tmp333f0r2j
	/tmp/tmphr356z5r
	/tmp/tmporp4rag2
	/tmp/tmpy7sjqdsw
Temp files created by test/test_datequery.py
Temp files created by test/test_dbcore.py
Temp files created by test/test_files.py
Temp files created by test/test_hidden.py
Temp files created by test/test_importer.py
	/tmp/tmp0m363gfb
	/tmp/tmp2n3i13mc
	/tmp/tmpxk3v304s
Temp files created by test/test_library.py
Temp files created by test/test_logging.py
Temp files created by test/test_m3ufile.py
Temp files created by test/test_mb.py
Temp files created by test/test_metasync.py
Temp files created by test/test_pipeline.py
Temp files created by test/test_plugins.py
	/tmp/tmp6pxhx67u
	/tmp/tmpb8pqi9ui
	/tmp/tmpcx_658g7
	/tmp/tmp_giqb9jz
	/tmp/tmpgm9xk94_
	/tmp/tmpk60l6bt3
	/tmp/tmpqoj4la68
	/tmp/tmptcdu20rp
	/tmp/tmpvr7k5shn
	/tmp/tmpwnfnzs91
Temp files created by test/test_query.py
Temp files created by test/test_sort.py
Temp files created by test/test_template.py
Temp files created by test/test_ui_commands.py
	/tmp/tmpns2u94w6
Temp files created by test/test_ui_importer.py
Temp files created by test/test_ui_init.py
Temp files created by test/test_ui.py
Temp files created by test/test_util.py
Temp files created by test/test_vfs.py

And that's what we have right now:

Temp files created by test/__init__.py
Temp files created by test/plugins/__init__.py
Temp files created by test/plugins/lyrics_download_samples.py
Temp files created by test/plugins/test_acousticbrainz.py
Temp files created by test/plugins/test_advancedrewrite.py
Temp files created by test/plugins/test_albumtypes.py
Temp files created by test/plugins/test_art.py
Temp files created by test/plugins/test_aura.py
Temp files created by test/plugins/test_bareasc.py
Temp files created by test/plugins/test_beatport.py
Temp files created by test/plugins/test_bucket.py
Temp files created by test/plugins/test_convert.py
Temp files created by test/plugins/test_discogs.py
Temp files created by test/plugins/test_edit.py
Temp files created by test/plugins/test_embedart.py
Temp files created by test/plugins/test_embyupdate.py
Temp files created by test/plugins/test_export.py
Temp files created by test/plugins/test_fetchart.py
Temp files created by test/plugins/test_filefilter.py
Temp files created by test/plugins/test_ftintitle.py
Temp files created by test/plugins/test_hook.py
Temp files created by test/plugins/test_ihate.py
Temp files created by test/plugins/test_importadded.py
Temp files created by test/plugins/test_importfeeds.py
Temp files created by test/plugins/test_info.py
Temp files created by test/plugins/test_ipfs.py
Temp files created by test/plugins/test_keyfinder.py
Temp files created by test/plugins/test_lastgenre.py
Temp files created by test/plugins/test_limit.py
Temp files created by test/plugins/test_lyrics.py
Temp files created by test/plugins/test_mbsubmit.py
Temp files created by test/plugins/test_mbsync.py
Temp files created by test/plugins/test_mpdstats.py
Temp files created by test/plugins/test_parentwork.py
Temp files created by test/plugins/test_permissions.py
Temp files created by test/plugins/test_player.py
Temp files created by test/plugins/test_playlist.py
Temp files created by test/plugins/test_play.py
Temp files created by test/plugins/test_plexupdate.py
Temp files created by test/plugins/test_plugin_mediafield.py
Temp files created by test/plugins/test_random.py
Temp files created by test/plugins/test_replaygain.py
Temp files created by test/plugins/test_smartplaylist.py
Temp files created by test/plugins/test_spotify.py
Temp files created by test/plugins/test_subsonicupdate.py
Temp files created by test/plugins/test_the.py
Temp files created by test/plugins/test_thumbnails.py
Temp files created by test/plugins/test_types_plugin.py
Temp files created by test/plugins/test_web.py
Temp files created by test/plugins/test_zero.py
Temp files created by test/rsrc/beetsplug/test.py
Temp files created by test/rsrc/convert_stub.py
Temp files created by test/testall.py
Temp files created by test/test_art_resize.py
Temp files created by test/test_autotag.py
Temp files created by test/test_config_command.py
Temp files created by test/test_datequery.py
Temp files created by test/test_dbcore.py
Temp files created by test/test_files.py
Temp files created by test/test_hidden.py
Temp files created by test/test_importer.py
Temp files created by test/test_library.py
Temp files created by test/test_logging.py
Temp files created by test/test_m3ufile.py
Temp files created by test/test_mb.py
Temp files created by test/test_metasync.py
Temp files created by test/test_pipeline.py
Temp files created by test/test_plugins.py
Temp files created by test/test_query.py
Temp files created by test/test_sort.py
Temp files created by test/test_template.py
Temp files created by test/test_ui_commands.py
Temp files created by test/test_ui_importer.py
Temp files created by test/test_ui_init.py
Temp files created by test/test_ui.py
Temp files created by test/test_util.py
Temp files created by test/test_vfs.py

Note that the command which provides the output is now available through poe.

@snejus snejus self-assigned this Jul 2, 2024
@snejus snejus requested a review from Serene-Arc July 2, 2024 14:26
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 2, 2024

Thank you for the PR! The changelog has not been updated, so here is a friendly reminder to check if you need to add an entry.

@snejus
test_art, test_embedart: Define FetchImageHelper without _common.Test…
56d9d96 
…Case

And move the definition to a shared module.

The problem was that EmbedartCliTest ran `_common.TestCase.setUp`
method which initialised temporary directory for the tests AND ran
`helper.TestHelper.setup_beets` method which initialised another set of
temporary directories. This meant that the first set of directories
could not be tracked down for the cleanup.
@snejus
fetchart, artresizer: Create art files in predictable directories
1fda7b6 
This allows to clean them up in art (1) fetching, (2) resizing and (3)
deinterlace tests.
@snejus
TestHelper: Create mediafield fixtures within temp_dir
c904521 
This way they get automatically removed with removal of temp_dir.

After this change `test/plugins/test_zero.py` does not any more leave
any mediafield fixtures hanging around.
@snejus
test_play: Remove files generated by play plugin
0682d0d
@snejus
test_config_command: use TestHelper to manage temp dir
12730aa 
Due to some weird race conditions the temporary directories were not
getting torn down for four of the tests. I failed to figure out why, and
I found that using TestHelper to manage the temporary directory somehow
fixes this.
@snejus
test_plugins.py: dedupe setUp, tearDown usage
f6b45f5 
And most importantly, remove a redudant invocation of `setup_beets`
which left temporary directories hanging around without getting cleaned
up.
@snejus
test_importer.py: do not run _common.TestCase.setUp with setup_beets
812bcb9
@snejus
test_ui_commands.py: make sure tearDown is run for FieldsTest
df08555
@snejus
Add a shell command which tests for leftover temp files
83a4d71
@snejus snejus mentioned this pull request Jul 2, 2024
Open
@Serene-Arc
Copy link
Contributor

Another reason why the switch to pytest would be a great thing. I'll give a review but what is the relation to the other PR, #5285? Do they conflict or are they about different classes?

@snejus
Copy link
Member Author

snejus commented Jul 3, 2024

I think this supersedes #5285

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Serene-Arc Serene-Arc Awaiting requested review from Serene-Arc

Assignees

@snejus snejus

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Ensure that tests do not leave any files behind
2 participants
@snejus @Serene-Arc