Purge process, some errors and suspected problems

Sure problem:

• Some projects with an active demand still appear in the "purgeable" list.

Maybe a problem:

• Make sure we don't check ".log.gz" files when looking for purgeable analyzes.
• Make sure the "purgeable" projects really do have expired files in their runs/analyzes.

changed milestone to %V3.4.1

added Bug Dysfunction labels

assigned to @rtherville

Regarding those:

• Make sure we don't check ".log.gz" files when looking for purgeable analyzes.
• Make sure the "purgeable" projects really do have expired files in their runs/analyzes.

Indeed, the "Log Files" analyzes, containing ".log.gz" files, keeps the "stored" status and appear again and again in the purgeable projects list. Some projects are shown as purgeable even though they only contain those logs.

About the wrong filter:

https://forgemia.inra.fr/genotoul-bioinfo/ng6/-/blob/master/ui/nG6/Classes/Controller/tx_nG6_db.php#L206

We need a way to exclude "Log Files" analyzes from this search (not great), or to mark them as 'purged' even if we keep the files during the purge process (sounds better).

https://forgemia.inra.fr/genotoul-bioinfo/ng6/-/blob/master/ui/nG6/Classes/Controller/tx_nG6_db.php#L206

It looks like the "Log Files" analyzes should already be set as "purged" without removing the files. Hmmmm. (So why do the analyzes stay 'stored' after the purge?)

As an other exemple, the following has been purged today : https://ng6.toulouse.inra.fr/projects?tx_nG6_pi1%5Bproject_id%5D=1873&tx_nG6_pi1%5Brun_id%5D=19585&cHash=510449465c6c419829c284eefa1459bb

The hidden analyzes have not been marked as "purged".

1st try:

I thought the following ssh2_exec() returned an error when the only files found were excluded from the purge (then, no purge, and no analysis.data_state -> "purged").

https://forgemia.inra.fr/genotoul-bioinfo/ng6/-/blob/master/ui/nG6/Classes/Controller/class.tx_nG6_utils.php#L152

Tested on the following:

Test sur "REPDEV" (7)
Run "Wabsarf1-dil-1-4" (17801)
Analyse "Log Files" (73007) /data/analyze/4176d30ee

And... Everything works fine. No file deleted, but analysis.data_state -> "purged".

2nd try:

Same thing, but analysis.hidden = 1.

And the problem occurs. Purge OK, but the analysis.data_state stays at 'stored'.

So the problem is not the ssh "find ... exec rm{}", it's that hidden analyzes are ignored when purging! (This explains a lot.)

But why? (We definitly need Gifs integration in forgemia!)

The purge only targets runs and analyzes listed in 'tx_nG6_purge_demand'. So, are the hidden analyzes in the demand, but excluded later? Or are they just absent from the demand?

The hidden analyses are just not added to the purge demand.

mentioned in commit bd50bd32

mentioned in merge request !144 (closed)

Commit tested, it works fine.

To close the subject, not only the hidden analyzes were impacted, but also hidden runs. We missed that bug, but the good news is that we'll free more space next month.

closed

reopened

removed milestone %V3.4.1

changed milestone to %V3.4.3

New problem to handle:

If an alert mail is supposed to be sent, but it only contains hidden runs/analyzes -> do not send any mail.

branch issue#240.2

mentioned in commit 1882dc56

Tested on ng6-test (Branch issue#240.2)

project 10XUmbra, 1 hidden run and 4 hidden analyzes

• The hidden elements appear in Admin -> Mail Obsolete Project, check!
• The hidden elements should appear in the DB demand (or they won't be purged), check!
• If the demand only concerns hidden elements, do not send any mail, check!
• During the purge, the hidden runs and analyzes should be purged, check!

mentioned in merge request !146 (merged)

added Unconformity label and removed Dysfunction label