Compare commits
12
Commits
54f16e0a20
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
 chris
|
69d038fe98 | ||
|
chris
|
5b811260a5 | ||
 chrisandClaude Sonnet 4.6
|
dfacd0ce83 | ||
|
chris
|
8234c2d9f3 | ||
|
chrisandClaude Sonnet 4.6
|
8bfc4d04b7 | ||
|
chris
|
eef48a9c8d | ||
|
chris
|
6677f4f0c2 | ||
|
chris
|
b33b859cf2 | ||
|
chris
|
44f143fbe8 | ||
|
chris
|
b1e02b737c | ||
|
chris
|
a8887aeeea | ||
|
chris
|
93e0576777 |
@@ -0,0 +1,226 @@
|
|||||||
|
# Bad-Signature-Toolkit
|
||||||
|
|
||||||
|
Werkzeuge zum Aufspüren und Beheben von **"Bad Signature"-Fehlern** der
|
||||||
|
Nextcloud Server-Side-Encryption (Dateien, die sich wegen einer
|
||||||
|
fehlerhaften Verschlüsselungs-Signatur nicht mehr öffnen lassen).
|
||||||
|
|
||||||
|
Alle Scripte laufen als **root** auf dem jeweiligen Nextcloud-Server und
|
||||||
|
fragen die Website interaktiv ab, falls `-s <website>` nicht angegeben
|
||||||
|
wird (Voraussetzung: eine passende `.conf`-Datei liegt im `conf/`-
|
||||||
|
Verzeichnis neben den Scripten).
|
||||||
|
|
||||||
|
## Der Ablauf auf einen Blick
|
||||||
|
|
||||||
|
```
|
||||||
|
1. scan_bad_signature.sh Betroffene Dateien FINDEN (read-only)
|
||||||
|
2. recover_bad_signature.sh Dateien probeweise ENTSCHLÜSSELN (read-only,
|
||||||
|
schreibt nur außerhalb von Nextcloud)
|
||||||
|
3. restore_bad_signature.sh Geprüfte Dateien ZURÜCKSCHREIBEN (live-Schreibzugriff)
|
||||||
|
4. recreate_bad_signature.sh Fallback für die Fälle, die (3) (live-Schreibzugriff,
|
||||||
|
nicht reparieren kann bricht Freigaben!)
|
||||||
|
delete_files.sh Explizite Dateiliste LÖSCHEN (live-Schreibzugriff,
|
||||||
|
(unabhängig vom obigen Ablauf) -> Papierkorb)
|
||||||
|
```
|
||||||
|
|
||||||
|
Jeder Schritt liest den Report des vorherigen Schritts. Reports landen
|
||||||
|
alle unter `reports/` (Dateiname verrät den Schritt: `bad_signature_…`,
|
||||||
|
`recovery_…`, `restore_…`, `recreate_…`, `delete_…`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. `scan_bad_signature.sh` – Betroffene Dateien finden
|
||||||
|
|
||||||
|
Liest einmal jede verschlüsselte Datei eines/mehrerer/aller Accounts und
|
||||||
|
protokolliert, welche dabei mit einem Signaturfehler scheitert. **Rein
|
||||||
|
lesend** – verändert, verschiebt oder löscht nichts.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scan_bad_signature.sh -s <website>
|
||||||
|
```
|
||||||
|
|
||||||
|
→ Ergebnis: `reports/bad_signature_<website>_<datum>.tsv`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. `recover_bad_signature.sh` – Versuchsweise entschlüsseln
|
||||||
|
|
||||||
|
Schaltet die Signaturprüfung *instanzweit, nur für die Laufzeit des
|
||||||
|
Scripts* ab (`encryption_skip_signature_check`) und versucht, die
|
||||||
|
betroffenen Dateien trotzdem zu entschlüsseln. Funktioniert nur, wenn
|
||||||
|
die Signaturprüfung selbst das Problem ist – nicht bei tatsächlich
|
||||||
|
beschädigtem Chiffretext. Jede gewonnene Datei wird automatisch geprüft
|
||||||
|
(Dateityp-Signatur, PDF-/ZIP-Integrität, Plausibilität der Dateigröße
|
||||||
|
o. Ä.) und als `VALID`, `UNVERIFIED` ("Nicht prüfbar") oder `INVALID`
|
||||||
|
("Datenmüll") eingestuft.
|
||||||
|
|
||||||
|
**Wichtig:** Original-Dateien in Nextcloud werden nie angefasst. Alles
|
||||||
|
landet in einem separaten Verzeichnis außerhalb von Nextclouds eigener
|
||||||
|
Ablage (`/var/nc-recovery/<website>/…`) – dort liegen danach
|
||||||
|
**unverschlüsselte** Daten, also nach Gebrauch aufräumen.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./recover_bad_signature.sh -s <website>
|
||||||
|
|
||||||
|
# Nur erneut validieren (z. B. nach einem Script-Update mit neuen
|
||||||
|
# Prüfregeln), ohne nochmal zu entschlüsseln:
|
||||||
|
./recover_bad_signature.sh -V -s <website>
|
||||||
|
```
|
||||||
|
|
||||||
|
→ Ergebnis: `reports/recovery_<website>_<datum>.tsv`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. `restore_bad_signature.sh` – Zurückschreiben (Regelfall)
|
||||||
|
|
||||||
|
Schreibt die in Schritt 2 gewonnenen Dateien über die **normale
|
||||||
|
Nextcloud-Files-API** an ihren ursprünglichen Pfad zurück – der einzige
|
||||||
|
Weg, der Datei-ID und bestehende Freigaben erhält. Danach ist die Datei
|
||||||
|
wieder ganz normal verschlüsselt, mit frischer, korrekter Signatur.
|
||||||
|
|
||||||
|
**Seit dem letzten Update werden standardmäßig zwei Kategorien
|
||||||
|
zurückgeschrieben**, klar unterscheidbar im Report (Spalte `origin`):
|
||||||
|
|
||||||
|
| validation im Recovery-Report | Bedeutung | wird restauriert |
|
||||||
|
|---|---|---|
|
||||||
|
| `VALID` | dedizierte Strukturprüfung hat den Inhalt bestätigt | ja |
|
||||||
|
| `UNVERIFIED`, Größe plausibel | kein dedizierter Check, aber nichts sieht falsch aus | ja |
|
||||||
|
| `UNVERIFIED`, Größe "LOOKS OFF" | Größenverhältnis auffällig – echtes Warnsignal | **nein** |
|
||||||
|
| `INVALID` ("Datenmüll") | Strukturprüfung fehlgeschlagen | **nein** |
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./restore_bad_signature.sh -s <website>
|
||||||
|
# nur Dry-Run (nichts wird geschrieben):
|
||||||
|
./restore_bad_signature.sh -n -s <website>
|
||||||
|
```
|
||||||
|
|
||||||
|
Fragt danach interaktiv: welcher Recovery-Report, welche(r) Account(s).
|
||||||
|
|
||||||
|
**Sicherheit:** Neuvalidierung direkt vor jedem Schreiben · aktueller
|
||||||
|
(noch kaputter) Chiffretext wird vorher byte-genau gesichert
|
||||||
|
(`/var/nc-restore-backup/<website>`) · jede Datei wird nach dem
|
||||||
|
Schreiben normal zurückgelesen und per SHA-256 verglichen · Dry-Run ist
|
||||||
|
Default, echter Lauf braucht eine ausdrückliche `YES`-Bestätigung.
|
||||||
|
|
||||||
|
⚠️ **Ein erneuter Lauf wählt wieder *alle* passenden Einträge des
|
||||||
|
gewählten Accounts aus dem Report** – auch bereits erfolgreich
|
||||||
|
restaurierte, nicht nur neue. Für ein gezieltes Nachziehen einzelner,
|
||||||
|
noch offener Dateien lieber einen auf diese Dateien reduzierten
|
||||||
|
Mini-Report verwenden, statt den ganzen Account-Bestand erneut zu
|
||||||
|
überschreiben.
|
||||||
|
|
||||||
|
→ Ergebnis: `reports/restore_<website>_<datum>.tsv`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. `recreate_bad_signature.sh` – Fallback für Schlüsselmaterial-Fehler
|
||||||
|
|
||||||
|
Nur nötig, wenn `restore_bad_signature.sh` bei einer Datei mit einem
|
||||||
|
**Schlüsselmaterial-Fehler** scheitert (`MultiKeyDecryptException` /
|
||||||
|
"probably this is a shared file…") statt mit "Bad Signature". Löscht
|
||||||
|
die kaputte Datei auf reiner Dateisystem-Ebene (inkl. altem
|
||||||
|
Schlüssel-Verzeichnis) und legt sie komplett neu an, mit frischem
|
||||||
|
Schlüssel.
|
||||||
|
|
||||||
|
⚠️ **Invasiver als restore:** Die Datei bekommt eine **neue Datei-ID**
|
||||||
|
– bestehende Freigaben, Kommentare, Tags und Versionshistorie dieser
|
||||||
|
Datei gehen dabei verloren und müssten danach manuell neu eingerichtet
|
||||||
|
werden. Nur verwenden, wenn Schritt 3 tatsächlich mit diesem
|
||||||
|
spezifischen Fehler gescheitert ist.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./recreate_bad_signature.sh -s <website>
|
||||||
|
# Dateien mit aktiven Freigaben standardmäßig übersprungen,
|
||||||
|
# nur mit Bedacht einschließen:
|
||||||
|
./recreate_bad_signature.sh -f -s <website>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sicherheit:** Nur Dateien mit passendem Fehler aus dem
|
||||||
|
restore-Report werden angefasst · Chiffretext UND Schlüssel-Verzeichnis
|
||||||
|
werden vorher gesichert und die Sicherung vor dem Löschen verifiziert
|
||||||
|
(`/var/nc-recreate-backup/<website>`) · Dateien mit aktiven Freigaben
|
||||||
|
werden standardmäßig übersprungen · Verifikation nach dem Schreiben wie
|
||||||
|
bei restore.
|
||||||
|
|
||||||
|
→ Ergebnis: `reports/recreate_<website>_<datum>.tsv`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `delete_files.sh` – Gezielt nicht benötigte Dateien löschen
|
||||||
|
|
||||||
|
Generisches, von Account und Site unabhängiges Script für den zweiten
|
||||||
|
Teil des Aufräum-Workflows: eine **von Hand geprüfte** Liste an
|
||||||
|
Dateien entfernen, die nicht erhaltenswert sind (z. B. macOS-Spotlight-
|
||||||
|
Indexdateien, Fragmente aus "Webseite speichern"). Kein Scannen, kein
|
||||||
|
automatisches Erraten – jede Zeile in der Liste ist eine bewusste
|
||||||
|
Entscheidung.
|
||||||
|
|
||||||
|
Pfadliste: einfache Textdatei, ein Pfad pro Zeile (`#`-Kommentare und
|
||||||
|
Leerzeilen erlaubt), Account wird automatisch aus dem Pfad erkannt:
|
||||||
|
|
||||||
|
```
|
||||||
|
/inge/files/Ordner/Datei.ext
|
||||||
|
/anderer-account/files/Anderer/Pfad/datei2.ext
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./delete_files.sh -s <website> -f <pfadliste.txt>
|
||||||
|
# nur Dry-Run:
|
||||||
|
./delete_files.sh -n -s <website> -f <pfadliste.txt>
|
||||||
|
```
|
||||||
|
|
||||||
|
Löscht über die normale Files-API (→ **Papierkorb**, sofern
|
||||||
|
`files_trashbin` aktiv ist) und sichert den Chiffretext zusätzlich
|
||||||
|
byte-genau vorher (`/var/nc-delete-backup/<website>`). Ein bereits
|
||||||
|
nicht mehr existierender Pfad wird als `NOT_FOUND` gemeldet, nicht als
|
||||||
|
Fehler.
|
||||||
|
|
||||||
|
→ Ergebnis: `reports/delete_<website>_<datum>.tsv`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Typischer Ablauf
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scan_bad_signature.sh -s cloud-01.oopen.de # betroffene Dateien finden
|
||||||
|
./recover_bad_signature.sh -s cloud-01.oopen.de # entschlüsseln + prüfen
|
||||||
|
./restore_bad_signature.sh -s cloud-01.oopen.de # VALID + plausible zurückschreiben
|
||||||
|
|
||||||
|
# nur bei einzelnen WRITE_ERROR mit Schlüsselmaterial-Fehler nötig:
|
||||||
|
./recreate_bad_signature.sh -s cloud-01.oopen.de
|
||||||
|
|
||||||
|
# optional: von Hand geprüfte, nicht erhaltenswerte Dateien entfernen
|
||||||
|
./delete_files.sh -s cloud-01.oopen.de -f nicht_benoetigt.txt
|
||||||
|
|
||||||
|
# zur Kontrolle: sollte jetzt (für erledigte Accounts) 0 melden
|
||||||
|
./scan_bad_signature.sh -s cloud-01.oopen.de
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verzeichnisse
|
||||||
|
|
||||||
|
| Zweck | Standardpfad | Override (conf-Datei) |
|
||||||
|
|---|---|---|
|
||||||
|
| Recovery-Kopien (unverschlüsselt!) | `/var/nc-recovery/<website>` | `RECOVERY_BASE_DIR` |
|
||||||
|
| Backup vor restore-Überschreiben | `/var/nc-restore-backup/<website>` | `RESTORE_BACKUP_BASE_DIR` |
|
||||||
|
| Backup vor recreate-Löschen | `/var/nc-recreate-backup/<website>` | `RECREATE_BACKUP_BASE_DIR` |
|
||||||
|
| Backup vor delete_files-Löschen | `/var/nc-delete-backup/<website>` | `DELETE_BACKUP_BASE_DIR` |
|
||||||
|
| Reports aller Scripte | `reports/` (neben den Scripten) | – |
|
||||||
|
|
||||||
|
## Sicherheitsprinzipien (gelten für alle schreibenden Scripte)
|
||||||
|
|
||||||
|
- **Dry-Run ist Standard** – ein echter Lauf braucht eine ausdrückliche `YES`-Bestätigung.
|
||||||
|
- **Immer zuerst sichern**, dann erst schreiben/löschen – byte-genaue Kopie, unabhängig von Nextcloud.
|
||||||
|
- **Immer neu validieren** unmittelbar vor dem Zugriff, nicht blind aus einem alten Report übernehmen.
|
||||||
|
- **Immer verifizieren** nach dem Schreiben (Rücklesen + SHA-256-Vergleich).
|
||||||
|
- Jedes Script kann per `Strg-C` sicher unterbrochen werden – kein halb geschriebener Report.
|
||||||
|
|
||||||
|
## Kurzreferenz aller Flags
|
||||||
|
|
||||||
|
| Script | `-s` | weitere Optionen |
|
||||||
|
|---|---|---|
|
||||||
|
| `scan_bad_signature.sh` | Website | – |
|
||||||
|
| `recover_bad_signature.sh` | Website | `-V` nur revalidieren |
|
||||||
|
| `restore_bad_signature.sh` | Website | `-n` Dry-Run erzwingen |
|
||||||
|
| `recreate_bad_signature.sh` | Website | `-n` Dry-Run erzwingen · `-f` Dateien mit aktiven Freigaben einschließen |
|
||||||
|
| `delete_files.sh` | Website | `-n` Dry-Run erzwingen · `-f <datei>` Pfadliste (**Pflicht**) |
|
||||||
|
|
||||||
|
`-h` zeigt bei jedem Script die ausführliche Hilfe.
|
||||||
@@ -0,0 +1,246 @@
|
|||||||
|
# Bad-Signature-Toolkit
|
||||||
|
|
||||||
|
Werkzeuge zum Aufspüren und Beheben von **"Bad Signature"-Fehlern** der
|
||||||
|
Nextcloud Server-Side-Encryption (Dateien, die sich wegen einer
|
||||||
|
fehlerhaften Verschlüsselungs-Signatur nicht mehr öffnen lassen).
|
||||||
|
|
||||||
|
Alle Scripte laufen als **root** auf dem jeweiligen Nextcloud-Server und
|
||||||
|
können auch **ganz ohne Parameter** aufgerufen werden – `-s <website>`
|
||||||
|
ist optional, nicht Pflicht. Fehlt es, ermittelt das Script alle
|
||||||
|
konfigurierten Instanzen anhand der `.conf`-Dateien im `conf/`-
|
||||||
|
Verzeichnis neben den Scripten. Gibt es davon **mehrere** (mehrere
|
||||||
|
Nextcloud-Instanzen auf demselben Server), wird interaktiv eine
|
||||||
|
Auswahlliste angezeigt, aus der die gewünschte Website ausgewählt wird.
|
||||||
|
Das gilt für alle fünf Scripte gleichermaßen.
|
||||||
|
|
||||||
|
## Der Ablauf auf einen Blick
|
||||||
|
|
||||||
|
```
|
||||||
|
1. scan_bad_signature.sh Betroffene Dateien FINDEN (read-only)
|
||||||
|
2. recover_bad_signature.sh Dateien probeweise ENTSCHLÜSSELN (read-only,
|
||||||
|
schreibt nur außerhalb von Nextcloud)
|
||||||
|
3. restore_bad_signature.sh Geprüfte Dateien ZURÜCKSCHREIBEN (live-Schreibzugriff)
|
||||||
|
4. recreate_bad_signature.sh Fallback für die Fälle, die (3) (live-Schreibzugriff,
|
||||||
|
nicht reparieren kann bricht Freigaben!)
|
||||||
|
delete_files.sh Explizite Dateiliste LÖSCHEN (live-Schreibzugriff,
|
||||||
|
(unabhängig vom obigen Ablauf) -> Papierkorb)
|
||||||
|
```
|
||||||
|
|
||||||
|
Jeder Schritt liest den Report des vorherigen Schritts. Reports landen
|
||||||
|
alle unter `reports/` (Dateiname verrät den Schritt: `bad_signature_…`,
|
||||||
|
`recovery_…`, `restore_…`, `recreate_…`, `delete_…`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. `scan_bad_signature.sh` – Betroffene Dateien finden
|
||||||
|
|
||||||
|
Liest einmal jede verschlüsselte Datei eines/mehrerer/aller Accounts und
|
||||||
|
protokolliert, welche dabei mit einem Signaturfehler scheitert. **Rein
|
||||||
|
lesend** – verändert, verschiebt oder löscht nichts.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scan_bad_signature.sh -s <website>
|
||||||
|
# oder ganz ohne Parameter - Website wird interaktiv abgefragt
|
||||||
|
# (bei mehreren konfigurierten Instanzen als Auswahlliste):
|
||||||
|
./scan_bad_signature.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
→ Ergebnis: `reports/bad_signature_<website>_<datum>.tsv`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. `recover_bad_signature.sh` – Versuchsweise entschlüsseln
|
||||||
|
|
||||||
|
Schaltet die Signaturprüfung *instanzweit, nur für die Laufzeit des
|
||||||
|
Scripts* ab (`encryption_skip_signature_check`) und versucht, die
|
||||||
|
betroffenen Dateien trotzdem zu entschlüsseln. Funktioniert nur, wenn
|
||||||
|
die Signaturprüfung selbst das Problem ist – nicht bei tatsächlich
|
||||||
|
beschädigtem Chiffretext. Jede gewonnene Datei wird automatisch geprüft
|
||||||
|
(Dateityp-Signatur, PDF-/ZIP-Integrität, Plausibilität der Dateigröße
|
||||||
|
o. Ä.) und als `VALID`, `UNVERIFIED` ("Nicht prüfbar") oder `INVALID`
|
||||||
|
("Datenmüll") eingestuft.
|
||||||
|
|
||||||
|
**Wichtig:** Original-Dateien in Nextcloud werden nie angefasst. Alles
|
||||||
|
landet in einem separaten Verzeichnis außerhalb von Nextclouds eigener
|
||||||
|
Ablage (`/var/nc-recovery/<website>/…`) – dort liegen danach
|
||||||
|
**unverschlüsselte** Daten, also nach Gebrauch aufräumen.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./recover_bad_signature.sh -s <website>
|
||||||
|
# oder ganz ohne Parameter - Website wird interaktiv abgefragt:
|
||||||
|
./recover_bad_signature.sh
|
||||||
|
|
||||||
|
# Nur erneut validieren (z. B. nach einem Script-Update mit neuen
|
||||||
|
# Prüfregeln), ohne nochmal zu entschlüsseln:
|
||||||
|
./recover_bad_signature.sh -V -s <website>
|
||||||
|
```
|
||||||
|
|
||||||
|
→ Ergebnis: `reports/recovery_<website>_<datum>.tsv`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. `restore_bad_signature.sh` – Zurückschreiben (Regelfall)
|
||||||
|
|
||||||
|
Schreibt die in Schritt 2 gewonnenen Dateien über die **normale
|
||||||
|
Nextcloud-Files-API** an ihren ursprünglichen Pfad zurück – der einzige
|
||||||
|
Weg, der Datei-ID und bestehende Freigaben erhält. Danach ist die Datei
|
||||||
|
wieder ganz normal verschlüsselt, mit frischer, korrekter Signatur.
|
||||||
|
|
||||||
|
**Seit dem letzten Update werden standardmäßig zwei Kategorien
|
||||||
|
zurückgeschrieben**, klar unterscheidbar im Report (Spalte `origin`):
|
||||||
|
|
||||||
|
| validation im Recovery-Report | Bedeutung | wird restauriert |
|
||||||
|
|---|---|---|
|
||||||
|
| `VALID` | dedizierte Strukturprüfung hat den Inhalt bestätigt | ja |
|
||||||
|
| `UNVERIFIED`, Größe plausibel | kein dedizierter Check, aber nichts sieht falsch aus | ja |
|
||||||
|
| `UNVERIFIED`, Größe "LOOKS OFF" | Größenverhältnis auffällig – echtes Warnsignal | **nein** |
|
||||||
|
| `INVALID` ("Datenmüll") | Strukturprüfung fehlgeschlagen | **nein** |
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./restore_bad_signature.sh -s <website>
|
||||||
|
# oder ganz ohne Parameter - Website wird interaktiv abgefragt:
|
||||||
|
./restore_bad_signature.sh
|
||||||
|
# nur Dry-Run (nichts wird geschrieben):
|
||||||
|
./restore_bad_signature.sh -n -s <website>
|
||||||
|
```
|
||||||
|
|
||||||
|
Fragt danach interaktiv: welcher Recovery-Report, welche(r) Account(s).
|
||||||
|
|
||||||
|
**Sicherheit:** Neuvalidierung direkt vor jedem Schreiben · aktueller
|
||||||
|
(noch kaputter) Chiffretext wird vorher byte-genau gesichert
|
||||||
|
(`/var/nc-restore-backup/<website>`) · jede Datei wird nach dem
|
||||||
|
Schreiben normal zurückgelesen und per SHA-256 verglichen · Dry-Run ist
|
||||||
|
Default, echter Lauf braucht eine ausdrückliche `YES`-Bestätigung.
|
||||||
|
|
||||||
|
⚠️ **Ein erneuter Lauf wählt wieder *alle* passenden Einträge des
|
||||||
|
gewählten Accounts aus dem Report** – auch bereits erfolgreich
|
||||||
|
restaurierte, nicht nur neue. Für ein gezieltes Nachziehen einzelner,
|
||||||
|
noch offener Dateien lieber einen auf diese Dateien reduzierten
|
||||||
|
Mini-Report verwenden, statt den ganzen Account-Bestand erneut zu
|
||||||
|
überschreiben.
|
||||||
|
|
||||||
|
→ Ergebnis: `reports/restore_<website>_<datum>.tsv`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. `recreate_bad_signature.sh` – Fallback für Schlüsselmaterial-Fehler
|
||||||
|
|
||||||
|
Nur nötig, wenn `restore_bad_signature.sh` bei einer Datei mit einem
|
||||||
|
**Schlüsselmaterial-Fehler** scheitert (`MultiKeyDecryptException` /
|
||||||
|
"probably this is a shared file…") statt mit "Bad Signature". Löscht
|
||||||
|
die kaputte Datei auf reiner Dateisystem-Ebene (inkl. altem
|
||||||
|
Schlüssel-Verzeichnis) und legt sie komplett neu an, mit frischem
|
||||||
|
Schlüssel.
|
||||||
|
|
||||||
|
⚠️ **Invasiver als restore:** Die Datei bekommt eine **neue Datei-ID**
|
||||||
|
– bestehende Freigaben, Kommentare, Tags und Versionshistorie dieser
|
||||||
|
Datei gehen dabei verloren und müssten danach manuell neu eingerichtet
|
||||||
|
werden. Nur verwenden, wenn Schritt 3 tatsächlich mit diesem
|
||||||
|
spezifischen Fehler gescheitert ist.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./recreate_bad_signature.sh -s <website>
|
||||||
|
# oder ganz ohne Parameter - Website wird interaktiv abgefragt:
|
||||||
|
./recreate_bad_signature.sh
|
||||||
|
# Dateien mit aktiven Freigaben standardmäßig übersprungen,
|
||||||
|
# nur mit Bedacht einschließen:
|
||||||
|
./recreate_bad_signature.sh -f -s <website>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sicherheit:** Nur Dateien mit passendem Fehler aus dem
|
||||||
|
restore-Report werden angefasst · Chiffretext UND Schlüssel-Verzeichnis
|
||||||
|
werden vorher gesichert und die Sicherung vor dem Löschen verifiziert
|
||||||
|
(`/var/nc-recreate-backup/<website>`) · Dateien mit aktiven Freigaben
|
||||||
|
werden standardmäßig übersprungen · Verifikation nach dem Schreiben wie
|
||||||
|
bei restore.
|
||||||
|
|
||||||
|
→ Ergebnis: `reports/recreate_<website>_<datum>.tsv`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `delete_files.sh` – Gezielt nicht benötigte Dateien löschen
|
||||||
|
|
||||||
|
Generisches, von Account und Site unabhängiges Script für den zweiten
|
||||||
|
Teil des Aufräum-Workflows: eine **von Hand geprüfte** Liste an
|
||||||
|
Dateien entfernen, die nicht erhaltenswert sind (z. B. macOS-Spotlight-
|
||||||
|
Indexdateien, Fragmente aus "Webseite speichern"). Kein Scannen, kein
|
||||||
|
automatisches Erraten – jede Zeile in der Liste ist eine bewusste
|
||||||
|
Entscheidung.
|
||||||
|
|
||||||
|
Pfadliste: einfache Textdatei, ein Pfad pro Zeile (`#`-Kommentare und
|
||||||
|
Leerzeilen erlaubt), Account wird automatisch aus dem Pfad erkannt:
|
||||||
|
|
||||||
|
```
|
||||||
|
/inge/files/Ordner/Datei.ext
|
||||||
|
/anderer-account/files/Anderer/Pfad/datei2.ext
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./delete_files.sh -s <website> -f <pfadliste.txt>
|
||||||
|
# oder ganz ohne Parameter - Website UND Pfad zur Liste werden
|
||||||
|
# interaktiv abgefragt:
|
||||||
|
./delete_files.sh
|
||||||
|
# nur Dry-Run:
|
||||||
|
./delete_files.sh -n -s <website> -f <pfadliste.txt>
|
||||||
|
```
|
||||||
|
|
||||||
|
Löscht über die normale Files-API (→ **Papierkorb**, sofern
|
||||||
|
`files_trashbin` aktiv ist) und sichert den Chiffretext zusätzlich
|
||||||
|
byte-genau vorher (`/var/nc-delete-backup/<website>`). Ein bereits
|
||||||
|
nicht mehr existierender Pfad wird als `NOT_FOUND` gemeldet, nicht als
|
||||||
|
Fehler.
|
||||||
|
|
||||||
|
→ Ergebnis: `reports/delete_<website>_<datum>.tsv`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Typischer Ablauf
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scan_bad_signature.sh -s cloud-01.oopen.de # betroffene Dateien finden
|
||||||
|
./recover_bad_signature.sh -s cloud-01.oopen.de # entschlüsseln + prüfen
|
||||||
|
./restore_bad_signature.sh -s cloud-01.oopen.de # VALID + plausible zurückschreiben
|
||||||
|
|
||||||
|
# nur bei einzelnen WRITE_ERROR mit Schlüsselmaterial-Fehler nötig:
|
||||||
|
./recreate_bad_signature.sh -s cloud-01.oopen.de
|
||||||
|
|
||||||
|
# optional: von Hand geprüfte, nicht erhaltenswerte Dateien entfernen
|
||||||
|
./delete_files.sh -s cloud-01.oopen.de -f nicht_benoetigt.txt
|
||||||
|
|
||||||
|
# zur Kontrolle: sollte jetzt (für erledigte Accounts) 0 melden
|
||||||
|
./scan_bad_signature.sh -s cloud-01.oopen.de
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verzeichnisse
|
||||||
|
|
||||||
|
| Zweck | Standardpfad | Override (conf-Datei) |
|
||||||
|
|---|---|---|
|
||||||
|
| Recovery-Kopien (unverschlüsselt!) | `/var/nc-recovery/<website>` | `RECOVERY_BASE_DIR` |
|
||||||
|
| Backup vor restore-Überschreiben | `/var/nc-restore-backup/<website>` | `RESTORE_BACKUP_BASE_DIR` |
|
||||||
|
| Backup vor recreate-Löschen | `/var/nc-recreate-backup/<website>` | `RECREATE_BACKUP_BASE_DIR` |
|
||||||
|
| Backup vor delete_files-Löschen | `/var/nc-delete-backup/<website>` | `DELETE_BACKUP_BASE_DIR` |
|
||||||
|
| Reports aller Scripte | `reports/` (neben den Scripten) | – |
|
||||||
|
|
||||||
|
## Sicherheitsprinzipien (gelten für alle schreibenden Scripte)
|
||||||
|
|
||||||
|
- **Dry-Run ist Standard** – ein echter Lauf braucht eine ausdrückliche `YES`-Bestätigung.
|
||||||
|
- **Immer zuerst sichern**, dann erst schreiben/löschen – byte-genaue Kopie, unabhängig von Nextcloud.
|
||||||
|
- **Immer neu validieren** unmittelbar vor dem Zugriff, nicht blind aus einem alten Report übernehmen.
|
||||||
|
- **Immer verifizieren** nach dem Schreiben (Rücklesen + SHA-256-Vergleich).
|
||||||
|
- Jedes Script kann per `Strg-C` sicher unterbrochen werden – kein halb geschriebener Report.
|
||||||
|
|
||||||
|
## Kurzreferenz aller Flags
|
||||||
|
|
||||||
|
`-s <website>` ist bei **allen** Scripten optional – fehlt es, wird die
|
||||||
|
Website interaktiv abgefragt (Auswahlliste bei mehreren konfigurierten
|
||||||
|
Instanzen).
|
||||||
|
|
||||||
|
| Script | `-s` (optional) | weitere Optionen |
|
||||||
|
|---|---|---|
|
||||||
|
| `scan_bad_signature.sh` | Website | – |
|
||||||
|
| `recover_bad_signature.sh` | Website | `-V` nur revalidieren |
|
||||||
|
| `restore_bad_signature.sh` | Website | `-n` Dry-Run erzwingen |
|
||||||
|
| `recreate_bad_signature.sh` | Website | `-n` Dry-Run erzwingen · `-f` Dateien mit aktiven Freigaben einschließen |
|
||||||
|
| `delete_files.sh` | Website | `-n` Dry-Run erzwingen · `-f <datei>` Pfadliste (ohne Angabe interaktiv abgefragt) |
|
||||||
|
|
||||||
|
`-h` zeigt bei jedem Script die ausführliche Hilfe.
|
||||||
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,134 @@
|
|||||||
|
# Bad-Signature-Toolkit: Zusammenfassung der neuen/aktualisierten Scripte
|
||||||
|
|
||||||
|
Stand: 14.09.2026
|
||||||
|
|
||||||
|
## Ausgangslage
|
||||||
|
|
||||||
|
Im Rahmen der Bereinigung von "Bad Signature"-Fehlern (fehlerhafte Server-Side-Encryption-Signaturen) auf mehreren Nextcloud-Instanzen wurde das bestehende Toolkit um zwei Punkte erweitert:
|
||||||
|
|
||||||
|
1. **`restore_bad_signature.sh`** schreibt jetzt standardmäßig nicht nur streng geprüfte ("VALID"), sondern auch plausible, aber nicht abschließend prüfbare Dateien zurück ("Nicht prüfbar"/UNVERIFIED).
|
||||||
|
2. **`delete_files.sh`** ist ein neues, generisches Script, um eine explizite Liste nicht benötigter Dateien zu löschen – unabhängig von Account oder Site, ohne Sonder-/Einmal-Scripte pro Vorfall.
|
||||||
|
|
||||||
|
Damit lässt sich der komplette Workflow konsequent nach einem einfachen Prinzip abbilden:
|
||||||
|
|
||||||
|
> **Unauffällige Dateien schreiben wir zurück. Dateien, die es nicht braucht, löschen wir.**
|
||||||
|
|
||||||
|
Beide Scripte reihen sich in das bestehende Toolkit ein:
|
||||||
|
|
||||||
|
```
|
||||||
|
scan_bad_signature.sh -> findet betroffene Dateien (read-only)
|
||||||
|
recover_bad_signature.sh -> entschlüsselt/prüft sie in ein separates Verzeichnis (read-only)
|
||||||
|
restore_bad_signature.sh -> schreibt geprüfte Dateien an ihren Originalort zurück
|
||||||
|
recreate_bad_signature.sh -> Fallback: löscht+erstellt neu (bricht Freigaben, nur für die schwierigere Fehlerklasse)
|
||||||
|
delete_files.sh -> NEU: löscht eine explizite Liste nicht benötigter Dateien
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. `restore_bad_signature.sh` (aktualisiert)
|
||||||
|
|
||||||
|
### Zweck (unverändert)
|
||||||
|
|
||||||
|
Schreibt bereits wiederhergestellte und geprüfte Dateien (Ergebnis eines vorherigen `recover_bad_signature.sh`-Laufs) über die normale Nextcloud-Files-API an ihren **ursprünglichen Pfad** zurück. Das ist der einzige Schreibweg, der Datei-ID und Freigaben erhält (im Gegensatz zu `recreate_bad_signature.sh`).
|
||||||
|
|
||||||
|
### Was sich geändert hat
|
||||||
|
|
||||||
|
Bisher wurden nur Dateien zurückgeschrieben, deren Recovery-Report-Eintrag `validation=VALID` trug – also Dateien, bei denen eine dedizierte Strukturprüfung (PDF-Header, ZIP-Integrität, Bildsignatur usw.) den Inhalt tatsächlich bestätigen konnte.
|
||||||
|
|
||||||
|
**Jetzt werden standardmäßig zwei Kategorien zurückgeschrieben:**
|
||||||
|
|
||||||
|
| Kategorie | Bedeutung | Wird restauriert? |
|
||||||
|
|---|---|---|
|
||||||
|
| `VALID` | Dedizierte Strukturprüfung hat den Inhalt bestätigt | Ja (wie bisher) |
|
||||||
|
| `UNVERIFIED` ("Nicht prüfbar"), Größenverhältnis plausibel | Kein dedizierter Check für diesen Dateityp vorhanden, aber nichts sieht falsch aus | **Ja, neu** |
|
||||||
|
| `UNVERIFIED`, Größenverhältnis "LOOKS OFF" | Wie oben, aber das Größenverhältnis der Datei ist auffällig | Nein – wie `INVALID`/"Datenmüll" ausgeschlossen |
|
||||||
|
| `INVALID` ("Datenmüll") | Strukturprüfung ist fehlgeschlagen | Nein (unverändert) |
|
||||||
|
|
||||||
|
Der einzige Fall, der tatsächlich ein Warnsignal ist (auffälliges Größenverhältnis), bleibt also weiterhin ausgeschlossen – kein Override möglich. Alles andere, das lediglich "nicht abschließend prüfbar, aber unauffällig" ist, wird ab sofort mit restauriert, ohne dass dafür ein Sonder-Script pro Account nötig wäre.
|
||||||
|
|
||||||
|
### Transparenz
|
||||||
|
|
||||||
|
Damit eine zurückgeschriebene UNVERIFIED-Datei nie mit einer echten VALID-Verifikation verwechselt wird:
|
||||||
|
|
||||||
|
- Der Restore-Report bekommt eine eigene Spalte `origin` (`VALID` oder `UNVERIFIED`).
|
||||||
|
- Eigene Zähler pro Account und in der Gesamtsumme ("davon 'Nicht prüfbar'/UNVERIFIED: N").
|
||||||
|
- Ausdrücklicher Hinweis im Report und am Ende des Laufs, diese Dateien stichprobenartig zu prüfen.
|
||||||
|
|
||||||
|
### Sicherheitsmechanismen (unverändert)
|
||||||
|
|
||||||
|
- Jede Datei wird unmittelbar vor dem Schreiben mit den *aktuellen* Prüfregeln erneut validiert (nicht blind aus dem alten Report übernommen).
|
||||||
|
- Der aktuell noch kaputte Chiffretext wird vor dem Überschreiben byte-genau gesichert (`/var/nc-restore-backup/<website>`).
|
||||||
|
- Nach jedem Schreiben wird die Datei über den normalen Lesepfad zurückgelesen und per SHA-256 mit der Quelle verglichen.
|
||||||
|
- Dry-Run ist Standard, ohne `-n` wird interaktiv gefragt; eine ausdrückliche `YES`-Bestätigung ist für den echten Lauf nötig.
|
||||||
|
|
||||||
|
### Aufruf
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./restore_bad_signature.sh -s <website>
|
||||||
|
# oder, nur Dry-Run:
|
||||||
|
./restore_bad_signature.sh -n -s <website>
|
||||||
|
```
|
||||||
|
|
||||||
|
Danach interaktiv: Auswahl des Recovery-Reports, Auswahl der Accounts ("VALID + plausibel UNVERIFIED"-Einträge).
|
||||||
|
|
||||||
|
> **Hinweis:** Ein erneuter Lauf über denselben Report wählt wieder *alle* passenden Einträge des jeweiligen Accounts aus – auch bereits erfolgreich restaurierte. Um gezielt nur einzelne, noch offene Dateien nachzuziehen (z. B. nachträglich als "unauffällig" eingestufte Dateien), empfiehlt es sich, einen auf diese Dateien reduzierten Mini-Report zu verwenden, statt den kompletten Account-Bestand erneut zu überschreiben.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. `delete_files.sh` (neu)
|
||||||
|
|
||||||
|
### Zweck
|
||||||
|
|
||||||
|
Generischer, wiederverwendbarer Begleiter zu `restore_bad_signature.sh` für die andere Hälfte des Workflows: eine explizite, von Hand geprüfte Liste von Dateien aus der Nextcloud-Live-Ablage entfernen – z. B. Systemdateien (macOS-Spotlight-Index), Fragmente aus "Webseite speichern"-Aktionen oder andere Dateien, die nach Durchsicht als nicht erhaltenswert eingestuft wurden.
|
||||||
|
|
||||||
|
**Nicht** account- oder site-spezifisch – ein einziges Script für alle Fälle, kein Einmal-Script pro Vorfall mehr nötig.
|
||||||
|
|
||||||
|
### Eingabeformat
|
||||||
|
|
||||||
|
Einfache Textdatei, ein Nextcloud-Pfad pro Zeile, im selben Format wie die `path`-Spalte der Toolkit-Reports:
|
||||||
|
|
||||||
|
```
|
||||||
|
/inge/files/Ordner/Datei.ext
|
||||||
|
# Kommentarzeilen mit '#' und Leerzeilen werden ignoriert
|
||||||
|
/anderer-account/files/Anderer/Pfad/datei2.ext
|
||||||
|
```
|
||||||
|
|
||||||
|
Der jeweilige Account wird automatisch aus dem ersten Pfadsegment erkannt – eine Liste kann also Dateien mehrerer Accounts derselben Site mischen.
|
||||||
|
|
||||||
|
### Was passiert beim Löschen
|
||||||
|
|
||||||
|
- Löschen erfolgt über die normale Nextcloud-Files-API (`$node->delete()`) – **derselbe Weg wie im Webinterface**. Ist die App `files_trashbin` aktiv (Standard), landet die Datei im Papierkorb des jeweiligen Accounts und ist von dort wiederherstellbar.
|
||||||
|
- Vor dem Löschen wird der aktuelle Chiffretext zusätzlich byte-genau in ein eigenes Backup-Verzeichnis gesichert (`/var/nc-delete-backup/<website>`) – unabhängig von Nextcloud/Papierkorb.
|
||||||
|
- Ein Pfad, der gar nicht mehr existiert, wird als `NOT_FOUND` gemeldet, nicht als Fehler.
|
||||||
|
- Dry-Run ist Standard, ohne `-n` wird interaktiv gefragt; eine ausdrückliche `YES`-Bestätigung ist für den echten Lauf nötig.
|
||||||
|
|
||||||
|
### Aufruf
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./delete_files.sh -s <website> -f <pfadliste.txt>
|
||||||
|
# oder, nur Dry-Run:
|
||||||
|
./delete_files.sh -n -s <website> -f <pfadliste.txt>
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Zusammenspiel der beiden Scripte
|
||||||
|
|
||||||
|
Für einen typischen "Nicht prüfbar"-Restbestand nach einem `recover_bad_signature.sh`-Lauf:
|
||||||
|
|
||||||
|
1. `restore_bad_signature.sh` läuft ganz normal für den Account – nimmt automatisch alle VALID- und plausiblen UNVERIFIED-Dateien mit.
|
||||||
|
2. Was danach als nicht erhaltenswert erkannt wird (z. B. Systemdateien), wandert in eine einfache Pfadliste.
|
||||||
|
3. `delete_files.sh` räumt diese Liste ab – unabhängig davon, ob die Dateien vorher restauriert wurden oder nicht.
|
||||||
|
|
||||||
|
Kein Account und kein Vorfall braucht dafür mehr ein eigenes Script – beide Werkzeuge sind allgemein einsetzbar und für jede Site/jeden Account nutzbar.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Beispiel aus der aktuellen Bereinigung (cloud-01.oopen.de)
|
||||||
|
|
||||||
|
Beim Recovery-Lauf für `cloud-01.oopen.de` (Accounts `chris` + `inge`) blieben 11 von 6249 Dateien als "Nicht prüfbar" übrig:
|
||||||
|
|
||||||
|
- **2 Dateien** (Ressourcen eines Lernspiels, `.ctf`/`.md8`) wurden als plausibel eingestuft und werden per `restore_bad_signature.sh` zurückgeschrieben.
|
||||||
|
- **9 Dateien** (7 macOS-Spotlight-Indexdateien, 2 Fragmente aus "Webseite speichern") wurden als nicht erhaltenswert eingestuft und werden per `delete_files.sh` entfernt.
|
||||||
|
|
||||||
|
Beide Scripte behandeln diesen Fall jetzt mit ihrer regulären, generischen Logik – ohne das ursprünglich dafür gebaute Einmal-Script `cleanup_11_misc_files_cloud01.sh`, das damit hinfällig ist.
|
||||||
Executable
+991
@@ -0,0 +1,991 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
CUR_IFS=$IFS
|
||||||
|
|
||||||
|
script_name="$(basename $(realpath $0))"
|
||||||
|
script_dir="$(dirname $(realpath $0))"
|
||||||
|
|
||||||
|
conf_dir="${script_dir}/conf"
|
||||||
|
snippet_dir="${script_dir}/snippets"
|
||||||
|
report_dir="${script_dir}/reports"
|
||||||
|
|
||||||
|
# - Before this script deletes a file from Nextcloud's live storage, it
|
||||||
|
# - keeps a raw, byte-for-byte copy of the CURRENT on-disk ciphertext
|
||||||
|
# - (the file exactly as it is right now) in a separate directory - a
|
||||||
|
# - plain filesystem copy, independent of Nextcloud/its Trash entirely.
|
||||||
|
# - This is an extra safety net on top of (not a replacement for)
|
||||||
|
# - Nextcloud's own Trash, which is where a normal delete through the
|
||||||
|
# - Files API already sends the file if the 'files_trashbin' app is
|
||||||
|
# - enabled.
|
||||||
|
# -
|
||||||
|
DEFAULT_DELETE_BACKUP_BASE_DIR="/var/nc-delete-backup"
|
||||||
|
|
||||||
|
declare -a unsorted_website_arr
|
||||||
|
declare -a website_arr
|
||||||
|
|
||||||
|
declare -a unsorted_account_arr
|
||||||
|
declare -a account_arr
|
||||||
|
|
||||||
|
LOCK_DIR="/tmp/${script_name%%.*}.LOCK"
|
||||||
|
log_file="${LOCK_DIR}/${script_name%%.*}.log"
|
||||||
|
|
||||||
|
run_date=$(date +%Y-%m-%d-%H%M)
|
||||||
|
|
||||||
|
|
||||||
|
# =============
|
||||||
|
# --- Some functions
|
||||||
|
# =============
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
|
||||||
|
[[ -n "$1" ]] && error "$1"
|
||||||
|
|
||||||
|
[[ $terminal ]] && echo -e "
|
||||||
|
\033[1mUsage:\033[m
|
||||||
|
|
||||||
|
$(basename $0) -s <website> -f <pathlist-file>
|
||||||
|
|
||||||
|
\033[1mDescription\033[m
|
||||||
|
|
||||||
|
Generic, reusable companion to restore_bad_signature.sh: deletes an
|
||||||
|
explicit, hand-picked list of files from a Nextcloud instance's live
|
||||||
|
storage, through Nextcloud's normal Files API - the same delete path
|
||||||
|
the web interface, WebDAV, or a sync client uses. If the
|
||||||
|
'files_trashbin' app is enabled (the default), the file ends up in
|
||||||
|
that account's Trash, not gone outright.
|
||||||
|
|
||||||
|
This is a plain, general-purpose tool - NOT specific to any one
|
||||||
|
account, site, or incident. It is meant for the second half of the
|
||||||
|
'bad signature' recovery workflow: after recover_bad_signature.sh
|
||||||
|
and restore_bad_signature.sh have taken care of every file that
|
||||||
|
validates cleanly or at least plausibly, a handful of files
|
||||||
|
sometimes remain that a human should look at and decide on one by
|
||||||
|
one (system/index files that don't belong in the cloud at all,
|
||||||
|
fragments from a browser 'save page' action, or anything else you
|
||||||
|
simply don't want kept). Once you've decided, list their paths in a
|
||||||
|
plain text file and point this script at it.
|
||||||
|
|
||||||
|
IMPORTANT:
|
||||||
|
- The path list file (-f) is plain text, one Nextcloud path per
|
||||||
|
line, in the SAME format used throughout this toolkit's own
|
||||||
|
reports (the 'path' column, e.g.
|
||||||
|
'/inge/files/Some/Folder/file.ext' - '/<uid>/files/...'). Blank
|
||||||
|
lines and lines starting with '#' are ignored. The account
|
||||||
|
(uid) is taken from each path's own first segment, so one list
|
||||||
|
can freely mix files from several accounts on the same site -
|
||||||
|
no separate account selection is needed.
|
||||||
|
- Nothing is guessed or auto-selected: every path this script
|
||||||
|
acts on is one YOU explicitly listed. There is no scanning, no
|
||||||
|
heuristic, no 'delete everything that looks like junk' mode
|
||||||
|
here - that judgment call happens before this script runs, not
|
||||||
|
inside it.
|
||||||
|
- Before each file is deleted (unless dry-run), the CURRENT
|
||||||
|
on-disk ciphertext is copied byte-for-byte to a separate backup
|
||||||
|
directory (default ${DEFAULT_DELETE_BACKUP_BASE_DIR}/<website>,
|
||||||
|
override with DELETE_BACKUP_BASE_DIR in the conf file) - a
|
||||||
|
plain filesystem copy that bypasses Nextcloud/encryption and
|
||||||
|
Trash entirely. This only works for local/default storage; it
|
||||||
|
is best-effort and a missing source is reported but does not
|
||||||
|
abort the run.
|
||||||
|
- The delete itself goes through Nextcloud's normal Files API
|
||||||
|
(\$node->delete()), so all of Nextcloud's own rules still
|
||||||
|
apply: if 'files_trashbin' is enabled the file is moved to that
|
||||||
|
account's Trash (recoverable there, subject to the instance's
|
||||||
|
own Trash retention settings); if it is disabled, the delete is
|
||||||
|
immediate and permanent.
|
||||||
|
- A path already absent (no matching file found) is reported as
|
||||||
|
NOT_FOUND, not as an error - nothing to do there.
|
||||||
|
|
||||||
|
Unless '-n' is given on the command line, you will be asked
|
||||||
|
interactively whether to run a dry-run (nothing is touched, just
|
||||||
|
reports what would happen) or the real thing.
|
||||||
|
|
||||||
|
\033[1mOptions\033[m
|
||||||
|
|
||||||
|
-s <website>
|
||||||
|
The site of the nextcloud instance.
|
||||||
|
|
||||||
|
-f <pathlist-file>
|
||||||
|
Plain text file with one Nextcloud path to delete per line (see
|
||||||
|
above). Required - if omitted, you will be asked for it.
|
||||||
|
|
||||||
|
-n
|
||||||
|
Dry-run: goes through selection and reporting, but does NOT back
|
||||||
|
up or delete anything. Passing it here skips the interactive mode
|
||||||
|
question mentioned above.
|
||||||
|
|
||||||
|
\033[1mExample:\033[m
|
||||||
|
|
||||||
|
Runs this script on system 'cloud-01.oopen.de'
|
||||||
|
|
||||||
|
$(basename $0) -s cloud-01.oopen.de -f /root/to_delete.txt
|
||||||
|
|
||||||
|
Dry-run only, nothing is deleted:
|
||||||
|
|
||||||
|
$(basename $0) -n -s cloud-01.oopen.de -f /root/to_delete.txt
|
||||||
|
"
|
||||||
|
|
||||||
|
clean_up 1
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
clean_up() {
|
||||||
|
|
||||||
|
# Perform program exit housekeeping
|
||||||
|
[[ -n "$delete_php_file" ]] && rm -f "$delete_php_file" 2> /dev/null
|
||||||
|
rm -rf "$LOCK_DIR"
|
||||||
|
blank_line
|
||||||
|
exit $1
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
echononl(){
|
||||||
|
if $terminal ; then
|
||||||
|
echo X\\c > /tmp/shprompt$$
|
||||||
|
if [ `wc -c /tmp/shprompt$$ | awk '{print $1}'` -eq 1 ]; then
|
||||||
|
echo -e -n "$*\\c" 1>&2
|
||||||
|
else
|
||||||
|
echo -e -n "$*" 1>&2
|
||||||
|
fi
|
||||||
|
rm /tmp/shprompt$$
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
echo_done() {
|
||||||
|
if $terminal ; then
|
||||||
|
echo -e "\033[75G[ \033[32mdone\033[m ]"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
echo_ok() {
|
||||||
|
if $terminal ; then
|
||||||
|
echo -e "\033[75G[ \033[32mok\033[m ]"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
echo_warning() {
|
||||||
|
if $terminal ; then
|
||||||
|
echo -e "\033[75G[ \033[33m\033[1mwarn\033[m ]"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
echo_failed(){
|
||||||
|
if $terminal ; then
|
||||||
|
echo -e "\033[75G[ \033[1;31mfailed\033[m ]"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
echo_skipped() {
|
||||||
|
if $terminal ; then
|
||||||
|
echo -e "\033[75G[ \033[37mskipped\033[m ]"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
fatal (){
|
||||||
|
echo ""
|
||||||
|
echo ""
|
||||||
|
if $terminal ; then
|
||||||
|
echo -e " [ \033[31m\033[1mFatal\033[m ]: \033[37m\033[1m$*\033[m"
|
||||||
|
echo ""
|
||||||
|
echo -e " \033[31m\033[1mScript will be interrupted..\033[m!"
|
||||||
|
else
|
||||||
|
echo " [ Fatal ]: $*"
|
||||||
|
echo ""
|
||||||
|
echo " Script was terminated...."
|
||||||
|
fi
|
||||||
|
clean_up 1
|
||||||
|
}
|
||||||
|
|
||||||
|
error(){
|
||||||
|
echo ""
|
||||||
|
if $terminal ; then
|
||||||
|
echo -e " [ \033[31m\033[1mError\033[m ]: $*"
|
||||||
|
else
|
||||||
|
echo " [ Error ]: $*"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
}
|
||||||
|
|
||||||
|
warn (){
|
||||||
|
if $terminal ; then
|
||||||
|
echo ""
|
||||||
|
echo -e " [ \033[33m\033[1mWarning\033[m ]: $*"
|
||||||
|
echo ""
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
info (){
|
||||||
|
if $terminal ; then
|
||||||
|
echo ""
|
||||||
|
echo -e " [ \033[32m\033[1mInfo\033[m ]: $*"
|
||||||
|
echo ""
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# - Remove leading/trailling whitespaces
|
||||||
|
# -
|
||||||
|
trim() {
|
||||||
|
local var="$*"
|
||||||
|
var="${var#"${var%%[![:space:]]*}"}" # remove leading whitespace characters
|
||||||
|
var="${var%"${var##*[![:space:]]}"}" # remove trailing whitespace characters
|
||||||
|
echo -n "$var"
|
||||||
|
}
|
||||||
|
|
||||||
|
## - Percentage (1 decimal) of parameter 1 (part) against parameter 2 (total)
|
||||||
|
## - Returns "0.0" if total is empty, zero or non-numeric.
|
||||||
|
## -
|
||||||
|
calc_percent() {
|
||||||
|
local _part="$1"
|
||||||
|
local _total="$2"
|
||||||
|
if [[ -z "$_total" ]] || ! [[ "$_total" =~ ^[0-9]+$ ]] || [[ "$_total" -eq 0 ]] ; then
|
||||||
|
echo "0.0"
|
||||||
|
else
|
||||||
|
awk -v b="$_part" -v t="$_total" 'BEGIN { printf "%.1f", (b/t)*100 }'
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
blank_line() {
|
||||||
|
if $terminal ; then
|
||||||
|
echo ""
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# - Running in a terminal?
|
||||||
|
# -
|
||||||
|
if [[ -t 1 ]] ; then
|
||||||
|
terminal=true
|
||||||
|
else
|
||||||
|
terminal=false
|
||||||
|
fi
|
||||||
|
|
||||||
|
# - This script needs root privileges (reads the Nextcloud data
|
||||||
|
# - directory for the raw ciphertext backup, 'su' into the webserver
|
||||||
|
# - user to perform the actual delete).
|
||||||
|
# -
|
||||||
|
if [[ "$(id -u)" -ne 0 ]] ; then
|
||||||
|
fatal "This script must be run as root (it needs to read the Nextcloud data directory for backups and 'su' into the webserver user). Please re-run as root, e.g. via sudo."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ----------
|
||||||
|
# - Jobhandling
|
||||||
|
# ----------
|
||||||
|
|
||||||
|
if pgrep -f "$(basename $0)" | grep -q -v $$ ; then
|
||||||
|
|
||||||
|
msg="A previos instance of script \"`basename $0`\" seems already be running."
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
if $terminal ; then
|
||||||
|
echo -e "[ \033[31m\033[1mFatal\033[m ]: $msg"
|
||||||
|
echo ""
|
||||||
|
echo -e " \033[31m\033[1mScript was interupted\033[m!"
|
||||||
|
else
|
||||||
|
echo " [ Fatal ]: $msg"
|
||||||
|
echo ""
|
||||||
|
echo " Script was interupted!"
|
||||||
|
fi
|
||||||
|
echo
|
||||||
|
|
||||||
|
exit 1
|
||||||
|
else
|
||||||
|
if [[ -d "$LOCK_DIR" ]] ; then
|
||||||
|
rm -rf "$LOCK_DIR" 2> /dev/null
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
||||||
|
# - If job already runs, stop execution..
|
||||||
|
# -
|
||||||
|
if mkdir "$LOCK_DIR" 2> /dev/null ; then
|
||||||
|
|
||||||
|
# - Remove lockdir when the script finishes, or when it receives a signal
|
||||||
|
# -
|
||||||
|
trap clean_up SIGHUP SIGINT SIGTERM
|
||||||
|
|
||||||
|
else
|
||||||
|
|
||||||
|
msg="A previos instance of script \"`basename $0`\" seems already be running."
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
if $terminal ; then
|
||||||
|
echo -e "[ \033[31m\033[1mFatal\033[m ]: $msg"
|
||||||
|
echo ""
|
||||||
|
echo -e " \033[31m\033[1mScript was interupted\033[m!"
|
||||||
|
else
|
||||||
|
echo " [ Fatal ]: $msg"
|
||||||
|
echo ""
|
||||||
|
echo " Script was interupted!"
|
||||||
|
fi
|
||||||
|
echo
|
||||||
|
|
||||||
|
exit 1
|
||||||
|
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
||||||
|
# -------------
|
||||||
|
# - Read in Commandline arguments
|
||||||
|
# -------------
|
||||||
|
DRY_RUN=false
|
||||||
|
_dry_run_explicit=false
|
||||||
|
PATHLIST_FILE=""
|
||||||
|
|
||||||
|
while getopts hns:f: opt ; do
|
||||||
|
case $opt in
|
||||||
|
h) usage ;;
|
||||||
|
n) DRY_RUN=true; _dry_run_explicit=true ;;
|
||||||
|
s) WEBSITE=$OPTARG ;;
|
||||||
|
f) PATHLIST_FILE=$OPTARG ;;
|
||||||
|
\?) usage
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
|
||||||
|
# =============
|
||||||
|
# --- Ask which mode to run in, unless '-n' was already given on the
|
||||||
|
# --- command line. The SAFE choice (dry-run) is the default here on
|
||||||
|
# --- purpose - this script deletes from live Nextcloud storage.
|
||||||
|
# =============
|
||||||
|
if ! $_dry_run_explicit && $terminal ; then
|
||||||
|
|
||||||
|
blank_line
|
||||||
|
echo -e "\033[37m\033[1mWhich mode should this run use?\033[m"
|
||||||
|
echo ""
|
||||||
|
echo -e " \033[1m[1] Dry-run\033[m - go through the list and report what would happen, but touch NOTHING"
|
||||||
|
echo ""
|
||||||
|
echo " [2] Real delete - actually back up and delete each listed file in Nextcloud"
|
||||||
|
info "Just press Return to use the default: [1] Dry-run."
|
||||||
|
echo -n " Select mode by number [1]: "
|
||||||
|
read _mode_choice
|
||||||
|
|
||||||
|
case "$(trim "$_mode_choice")" in
|
||||||
|
2) DRY_RUN=false ;;
|
||||||
|
""|1) DRY_RUN=true ;;
|
||||||
|
*) fatal "Invalid selection '$_mode_choice'." ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
||||||
|
if [[ -z "$PATHLIST_FILE" ]] ; then
|
||||||
|
blank_line
|
||||||
|
echo -n " Path to the file listing the Nextcloud paths to delete (-f): "
|
||||||
|
read PATHLIST_FILE
|
||||||
|
fi
|
||||||
|
|
||||||
|
PATHLIST_FILE="$(trim "$PATHLIST_FILE")"
|
||||||
|
|
||||||
|
if [[ -z "$PATHLIST_FILE" ]] ; then
|
||||||
|
fatal "No path list file given."
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ ! -f "$PATHLIST_FILE" ]] ; then
|
||||||
|
fatal "Path list file '$PATHLIST_FILE' not found."
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
||||||
|
if [[ -z "$WEBSITE" ]] ; then
|
||||||
|
|
||||||
|
while IFS='' read -r -d '' _conf_file ; do
|
||||||
|
source $_conf_file
|
||||||
|
if [[ -n "$WEBSITE" ]] ; then
|
||||||
|
unsorted_website_arr+=("${WEBSITE}:$_conf_file")
|
||||||
|
fi
|
||||||
|
WEBSITE=""
|
||||||
|
done < <(find "${conf_dir}" -maxdepth 1 -type f -name "*.conf" -print0)
|
||||||
|
|
||||||
|
# - Sort array
|
||||||
|
# -
|
||||||
|
IFS=$'\n' website_arr=($(sort <<<"${unsorted_website_arr[*]}"))
|
||||||
|
|
||||||
|
# Which cloud instance (website) would you like to update
|
||||||
|
#
|
||||||
|
source ${snippet_dir}/get-cloud-instance-to-update.sh
|
||||||
|
|
||||||
|
else
|
||||||
|
|
||||||
|
while IFS='' read -r -d '' _conf_file ; do
|
||||||
|
if $(grep -E -q "WEBSITE=\"?${WEBSITE}\"?" ${_conf_file} 2> /dev/null) ; then
|
||||||
|
conf_file="${_conf_file}"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
done < <(find "${conf_dir}" -maxdepth 1 -type f -name "*.conf" -print0)
|
||||||
|
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
||||||
|
# - Reset IFS
|
||||||
|
# -
|
||||||
|
IFS=$CUR_IFS
|
||||||
|
|
||||||
|
DEFAULT_SRC_BASE_DIR="/usr/local/src/nextcloud"
|
||||||
|
DEFAULT_HTTP_USER="www-data"
|
||||||
|
DEFAULT_HTTP_GROUP="www-data"
|
||||||
|
DEFAULT_PHP_ENGINE='FPM'
|
||||||
|
|
||||||
|
blank_line
|
||||||
|
echononl " Include Configuration file '$(basename "${conf_file}")'.."
|
||||||
|
if [[ ! -f $conf_file ]]; then
|
||||||
|
echo_skipped
|
||||||
|
fatal "Missing configuration file '$conf_file'."
|
||||||
|
else
|
||||||
|
source $conf_file
|
||||||
|
echo_ok
|
||||||
|
fi
|
||||||
|
|
||||||
|
DEFAULT_WEB_BASE_DIR="/var/www/${WEBSITE}"
|
||||||
|
[[ -n "$WEB_BASE_DIR" ]] || WEB_BASE_DIR=$DEFAULT_WEB_BASE_DIR
|
||||||
|
|
||||||
|
if [[ ! -d ${WEB_BASE_DIR} ]] ; then
|
||||||
|
fatal "Web base directory '$WEB_BASE_DIR' not found!"
|
||||||
|
fi
|
||||||
|
|
||||||
|
DATA_DIR="$(realpath ${WEB_BASE_DIR}/data)"
|
||||||
|
|
||||||
|
[[ -n "$PHP_ENGINE" ]] || PHP_ENGINE=$DEFAULT_PHP_ENGINE
|
||||||
|
|
||||||
|
INSTALL_DIR="$(realpath ${WEB_BASE_DIR}/nextcloud)"
|
||||||
|
CURRENT_VERSION="$(basename $INSTALL_DIR | cut -d"-" -f2)"
|
||||||
|
|
||||||
|
[[ -n "$DELETE_BACKUP_BASE_DIR" ]] || DELETE_BACKUP_BASE_DIR=$DEFAULT_DELETE_BACKUP_BASE_DIR
|
||||||
|
backup_dir="${DELETE_BACKUP_BASE_DIR}/${WEBSITE}"
|
||||||
|
|
||||||
|
|
||||||
|
# =============
|
||||||
|
# --- Some
|
||||||
|
# =============
|
||||||
|
|
||||||
|
# - Support systemd ?
|
||||||
|
# -
|
||||||
|
SYSTEMD_EXISTS=false
|
||||||
|
systemd=$(which systemd)
|
||||||
|
systemctl=$(which systemctl)
|
||||||
|
|
||||||
|
if [[ -n "$systemd" ]] || [[ -n "$systemctl" ]] ; then
|
||||||
|
SYSTEMD_EXISTS=true
|
||||||
|
fi
|
||||||
|
|
||||||
|
if $terminal ; then
|
||||||
|
echo ""
|
||||||
|
echo -e "\033[32m-----\033[m"
|
||||||
|
echo -e "Delete an explicit, hand-picked list of files from system \033[1m${WEB_BASE_DIR}\033[m"
|
||||||
|
echo -e "\033[1m
|
||||||
|
This deletes from Nextcloud's live storage (goes to that account's
|
||||||
|
Trash if 'files_trashbin' is enabled). Only the exact paths listed
|
||||||
|
in '${PATHLIST_FILE}' are ever touched - nothing is scanned,
|
||||||
|
guessed or auto-selected. The current on-disk ciphertext is backed
|
||||||
|
up first, independent of Nextcloud/Trash entirely.\033[m"
|
||||||
|
echo -e "\033[32m-----\033[m"
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
||||||
|
# =============
|
||||||
|
# --- Some checks
|
||||||
|
# =============
|
||||||
|
|
||||||
|
DEFAULT_HTTP_USER="www-data"
|
||||||
|
DEFAULT_HTTP_GROUP="www-data"
|
||||||
|
|
||||||
|
|
||||||
|
NGINX_IS_ENABLED=false
|
||||||
|
APACHE2_IS_ENABLED=false
|
||||||
|
|
||||||
|
# Get Webservice environment as IS_HTTPD_RUNNING, HTTP_USER, HTTP_GROUP..
|
||||||
|
#
|
||||||
|
source ${snippet_dir}/get-webservice-environment.sh
|
||||||
|
|
||||||
|
|
||||||
|
# Check PHP Version
|
||||||
|
#
|
||||||
|
source ${snippet_dir}/get-php-major-version.sh
|
||||||
|
|
||||||
|
|
||||||
|
# Get full qualified PHP command
|
||||||
|
#
|
||||||
|
source ${snippet_dir}/get-path-of-php-command.sh
|
||||||
|
|
||||||
|
|
||||||
|
if [[ ! -x "$PHP_BIN" ]]; then
|
||||||
|
fatal "No PHP binary found!"
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
||||||
|
# =============
|
||||||
|
# --- Read and parse the path list file: one Nextcloud path per line,
|
||||||
|
# --- blank lines and '#' comments ignored. The account (uid) for each
|
||||||
|
# --- path is taken from the path's own first segment
|
||||||
|
# --- ('/<uid>/files/...'), so one list can span several accounts.
|
||||||
|
# =============
|
||||||
|
|
||||||
|
parsed_entries_file="${LOCK_DIR}/parsed_entries.tsv"
|
||||||
|
declare -i total_malformed=0
|
||||||
|
declare -a malformed_lines=()
|
||||||
|
|
||||||
|
> "$parsed_entries_file"
|
||||||
|
while IFS= read -r _line || [[ -n "$_line" ]] ; do
|
||||||
|
_line="$(trim "$_line")"
|
||||||
|
[[ -z "$_line" ]] && continue
|
||||||
|
[[ "$_line" == \#* ]] && continue
|
||||||
|
|
||||||
|
if [[ "$_line" =~ ^/([^/]+)/files/ ]] ; then
|
||||||
|
_uid="${BASH_REMATCH[1]}"
|
||||||
|
printf '%s\t%s\n' "$_uid" "$_line" >> "$parsed_entries_file"
|
||||||
|
else
|
||||||
|
(( total_malformed++ ))
|
||||||
|
malformed_lines+=("$_line")
|
||||||
|
fi
|
||||||
|
done < "$PATHLIST_FILE"
|
||||||
|
|
||||||
|
if [[ ! -s "$parsed_entries_file" ]] ; then
|
||||||
|
fatal "No usable paths found in '$PATHLIST_FILE' (expected one '/<uid>/files/...' path per line)."
|
||||||
|
fi
|
||||||
|
|
||||||
|
unsorted_account_arr=($(cut -f1 "$parsed_entries_file" | sort -u))
|
||||||
|
IFS=$'\n' account_arr=($(sort <<<"${unsorted_account_arr[*]}"))
|
||||||
|
IFS=$CUR_IFS
|
||||||
|
|
||||||
|
blank_line
|
||||||
|
if $terminal ; then
|
||||||
|
echo -e "\033[37m\033[1mAccounts found in '$(basename "$PATHLIST_FILE")'\033[m"
|
||||||
|
echo ""
|
||||||
|
for _a in "${account_arr[@]}" ; do
|
||||||
|
_a_count=$(awk -F'\t' -v u="$_a" '$1==u' "$parsed_entries_file" | wc -l)
|
||||||
|
echo " - $_a (${_a_count})"
|
||||||
|
done
|
||||||
|
echo ""
|
||||||
|
if [[ $total_malformed -gt 0 ]] ; then
|
||||||
|
warn "${total_malformed} line(s) in '$PATHLIST_FILE' did not look like a '/<uid>/files/...' path and were IGNORED:"
|
||||||
|
for _l in "${malformed_lines[@]}" ; do
|
||||||
|
echo " $_l"
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
||||||
|
# =============
|
||||||
|
# --- Confirmation
|
||||||
|
# =============
|
||||||
|
|
||||||
|
mkdir -p "$report_dir" 2> /dev/null
|
||||||
|
delete_report_file="${report_dir}/delete_${WEBSITE}_${run_date}.tsv"
|
||||||
|
|
||||||
|
if $terminal ; then
|
||||||
|
echo ""
|
||||||
|
if $DRY_RUN ; then
|
||||||
|
echo -e "\033[1;32mStarting DRY-RUN delete for \033[1;37m${WEBSITE}\033[m"
|
||||||
|
else
|
||||||
|
echo -e "\033[1;31m\033[1mStarting REAL delete (removes from live Nextcloud storage) for \033[1;37m${WEBSITE}\033[m"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
echo -e " Cloud instance..........................: $WEBSITE"
|
||||||
|
echo -e " Path list file...........................: $PATHLIST_FILE"
|
||||||
|
echo -e " Accounts affected........................: \033[33m${account_arr[*]}\033[m"
|
||||||
|
echo ""
|
||||||
|
if $DRY_RUN ; then
|
||||||
|
echo -e " Nothing will be touched - dry-run only."
|
||||||
|
else
|
||||||
|
echo -e " Ciphertext backup (before delete)........: $backup_dir"
|
||||||
|
echo -e " Files deleted at their ORIGINAL path in Nextcloud (-> Trash, if enabled)."
|
||||||
|
fi
|
||||||
|
echo -e " Delete report.............................: reports/$(basename "${delete_report_file}")"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
if $DRY_RUN ; then
|
||||||
|
info "Dry-run: nothing is backed up or deleted."
|
||||||
|
else
|
||||||
|
warn "This DELETES from Nextcloud's live storage (through the normal Files API - goes to Trash if 'files_trashbin' is enabled). Only the exact paths listed in '${PATHLIST_FILE}' are touched. The current on-disk ciphertext is copied byte-for-byte to '${backup_dir}' first, independent of Nextcloud/Trash entirely."
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -n " Type upper case 'YES' to continue executing with this parameters: "
|
||||||
|
read OK
|
||||||
|
if [[ "$OK" = "YES" ]] ; then
|
||||||
|
echo ""
|
||||||
|
echo ""
|
||||||
|
echo -e "\033[1;32mGoing to delete the listed files for each affected account on \033[1;37m$WEBSITE \033[m"
|
||||||
|
else
|
||||||
|
fatal "Abort by user request - Answer as not 'YES'"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
||||||
|
{
|
||||||
|
echo "=================================================================="
|
||||||
|
echo " Delete-Versuch (Liste) ${WEBSITE} ${run_date}"
|
||||||
|
echo "=================================================================="
|
||||||
|
echo ""
|
||||||
|
echo " Pfadliste: $(basename "$PATHLIST_FILE")"
|
||||||
|
echo " Nur die dort explizit aufgefuehrten Pfade werden geloescht."
|
||||||
|
$DRY_RUN && echo " DRY-RUN: es wurde NICHTS gesichert oder geloescht."
|
||||||
|
! $DRY_RUN && echo " Chiffretext-Sicherung (vor dem Loeschen) liegt unter: ${backup_dir}"
|
||||||
|
[[ $total_malformed -gt 0 ]] && echo " ${total_malformed} Zeile(n) aus der Pfadliste wurden ignoriert (kein '/<uid>/files/...'-Pfad)."
|
||||||
|
echo ""
|
||||||
|
echo -e "path\tstatus\tdetail"
|
||||||
|
} > "$delete_report_file"
|
||||||
|
|
||||||
|
|
||||||
|
# =============
|
||||||
|
# --- Write the embedded PHP delete script
|
||||||
|
# =============
|
||||||
|
|
||||||
|
delete_php_file="${INSTALL_DIR}/.delete_files_$$.php"
|
||||||
|
|
||||||
|
cat > "$delete_php_file" <<'PHP_DELETE_EOF'
|
||||||
|
<?php
|
||||||
|
/**
|
||||||
|
* delete_files.php
|
||||||
|
*
|
||||||
|
* Deletes an explicit list of files from Nextcloud's own storage, at
|
||||||
|
* their ORIGINAL path, through the normal Files API ($node->delete())
|
||||||
|
* - the same delete path the web interface, WebDAV, or a sync client
|
||||||
|
* uses. If the 'files_trashbin' app is enabled, Nextcloud moves the
|
||||||
|
* file to that account's Trash itself; this script does not bypass
|
||||||
|
* that.
|
||||||
|
*
|
||||||
|
* Usage: php delete_files.php <uid> <listFile> [--dry-run]
|
||||||
|
* listFile: one Nextcloud path per line (no header, no other columns)
|
||||||
|
*/
|
||||||
|
|
||||||
|
error_reporting(E_ALL);
|
||||||
|
ini_set('display_errors', '1');
|
||||||
|
|
||||||
|
register_shutdown_function(function () {
|
||||||
|
$err = error_get_last();
|
||||||
|
if ($err !== null && in_array($err['type'], [E_ERROR, E_PARSE, E_CORE_ERROR, E_COMPILE_ERROR], true)) {
|
||||||
|
fwrite(STDERR, "DEBUG: FATAL bei Shutdown: [{$err['type']}] {$err['message']} in {$err['file']}:{$err['line']}\n");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
define('OC_CONSOLE', 1);
|
||||||
|
|
||||||
|
$oldWorkingDir = getcwd();
|
||||||
|
if ($oldWorkingDir === false) {
|
||||||
|
fwrite(STDERR, "Konnte aktuelles Arbeitsverzeichnis nicht ermitteln. Bitte mit absolutem Pfad aufrufen.\n");
|
||||||
|
exit(1);
|
||||||
|
}
|
||||||
|
chdir(__DIR__);
|
||||||
|
require_once __DIR__ . '/lib/base.php';
|
||||||
|
chdir($oldWorkingDir);
|
||||||
|
|
||||||
|
if (function_exists('posix_getuid') && posix_getuid() === 0) {
|
||||||
|
fwrite(STDERR, "Bitte NICHT als root ausfuehren, sondern als Webserver-User.\n");
|
||||||
|
exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
$uid = $argv[1] ?? null;
|
||||||
|
$listFile = $argv[2] ?? null;
|
||||||
|
$dryRun = in_array('--dry-run', $argv, true);
|
||||||
|
|
||||||
|
if ($uid === null || $listFile === null) {
|
||||||
|
fwrite(STDERR, "Usage: php delete_files.php <uid> <listFile> [--dry-run]\n");
|
||||||
|
exit(1);
|
||||||
|
}
|
||||||
|
if (!is_file($listFile)) {
|
||||||
|
fwrite(STDERR, "Liste nicht gefunden: $listFile\n");
|
||||||
|
exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
$rootFolder = \OC::$server->get(\OCP\Files\IRootFolder::class);
|
||||||
|
\OC_Util::setupFS($uid);
|
||||||
|
$userFolder = $rootFolder->getUserFolder($uid);
|
||||||
|
|
||||||
|
$lines = file($listFile, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
|
||||||
|
if ($lines === false) {
|
||||||
|
$lines = [];
|
||||||
|
}
|
||||||
|
|
||||||
|
$total = 0;
|
||||||
|
$deleted = 0;
|
||||||
|
$notFound = 0;
|
||||||
|
$failed = 0;
|
||||||
|
|
||||||
|
echo "path\tstatus\tdetail\n";
|
||||||
|
|
||||||
|
foreach ($lines as $path) {
|
||||||
|
$path = trim($path);
|
||||||
|
if ($path === '') {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
$total++;
|
||||||
|
|
||||||
|
try {
|
||||||
|
$node = null;
|
||||||
|
try {
|
||||||
|
$node = $rootFolder->get($path);
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
$node = null;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($node === null) {
|
||||||
|
$notFound++;
|
||||||
|
echo "$path\tNOT_FOUND\talready absent - nothing to delete\n";
|
||||||
|
flush();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($dryRun) {
|
||||||
|
$deleted++;
|
||||||
|
$size = ($node instanceof \OCP\Files\File) ? $node->getSize() : 0;
|
||||||
|
echo "$path\tDRY_RUN\twould delete now (${size} bytes) - goes to Trash if 'files_trashbin' is enabled\n";
|
||||||
|
flush();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
$node->delete();
|
||||||
|
|
||||||
|
// Verify: the normal read path should no longer find it
|
||||||
|
// (it may still exist in Trash - that is Nextcloud's own
|
||||||
|
// behaviour, not a failure of this delete).
|
||||||
|
clearstatcache();
|
||||||
|
$stillThere = null;
|
||||||
|
try {
|
||||||
|
$stillThere = $rootFolder->get($path);
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
$stillThere = null;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($stillThere === null) {
|
||||||
|
$deleted++;
|
||||||
|
echo "$path\tOK\tdeleted (moved to Trash, if 'files_trashbin' is enabled)\n";
|
||||||
|
} else {
|
||||||
|
$failed++;
|
||||||
|
echo "$path\tDELETE_ERROR\tdelete() did not throw, but the path is still resolvable afterwards\n";
|
||||||
|
}
|
||||||
|
flush();
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
$failed++;
|
||||||
|
$msg = str_replace(["\t", "\n", "\r"], ' ', $e->getMessage());
|
||||||
|
echo "$path\tDELETE_ERROR\t" . get_class($e) . ": $msg\n";
|
||||||
|
flush();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fwrite(STDERR, "\nFertig. Dateien verarbeitet: $total, geloescht: $deleted, nicht gefunden: $notFound, Fehler: $failed\n");
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
fwrite(STDERR, "\n!!! UNBEHANDELTE AUSNAHME !!!\n");
|
||||||
|
fwrite(STDERR, get_class($e) . ": " . $e->getMessage() . "\n");
|
||||||
|
fwrite(STDERR, "in " . $e->getFile() . ":" . $e->getLine() . "\n");
|
||||||
|
fwrite(STDERR, $e->getTraceAsString() . "\n");
|
||||||
|
exit(2);
|
||||||
|
}
|
||||||
|
PHP_DELETE_EOF
|
||||||
|
|
||||||
|
chmod 644 "$delete_php_file"
|
||||||
|
|
||||||
|
if ! $DRY_RUN ; then
|
||||||
|
mkdir -p "$backup_dir" 2> /dev/null
|
||||||
|
fi
|
||||||
|
|
||||||
|
declare -i total_selected=0
|
||||||
|
declare -i total_ok=0
|
||||||
|
declare -i total_not_found=0
|
||||||
|
declare -i total_errors=0
|
||||||
|
declare -i total_failed_users=0
|
||||||
|
|
||||||
|
for _user in "${account_arr[@]}" ; do
|
||||||
|
|
||||||
|
_sep_len=${#_user}
|
||||||
|
[[ $_sep_len -lt 9 ]] && _sep_len=9
|
||||||
|
_sep_line="$(printf '%*s' "$_sep_len" '' | tr ' ' '-')"
|
||||||
|
|
||||||
|
{
|
||||||
|
echo ""
|
||||||
|
echo ""
|
||||||
|
echo "$_sep_line"
|
||||||
|
echo "$_user"
|
||||||
|
echo "$_sep_line"
|
||||||
|
} >> "$delete_report_file"
|
||||||
|
|
||||||
|
declare -i _user_selected=0
|
||||||
|
|
||||||
|
list_file="${LOCK_DIR}/list_${_user}.tsv"
|
||||||
|
> "$list_file"
|
||||||
|
|
||||||
|
while IFS=$'\t' read -r _u _path ; do
|
||||||
|
[[ "$_u" != "$_user" ]] && continue
|
||||||
|
(( _user_selected++ ))
|
||||||
|
|
||||||
|
if ! $DRY_RUN ; then
|
||||||
|
_rel="${_path#/${_user}/files/}"
|
||||||
|
_orig_ciphertext="${DATA_DIR}/${_user}/files/${_rel}"
|
||||||
|
if [[ -f "$_orig_ciphertext" ]] ; then
|
||||||
|
_backup_target="${backup_dir}/${_user}/${_rel}"
|
||||||
|
mkdir -p "$(dirname "$_backup_target")" 2> /dev/null
|
||||||
|
cp -p "$_orig_ciphertext" "$_backup_target" 2> /dev/null
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
printf '%s\n' "$_path" >> "$list_file"
|
||||||
|
|
||||||
|
done < "$parsed_entries_file"
|
||||||
|
|
||||||
|
total_selected+=$_user_selected
|
||||||
|
|
||||||
|
user_result_tsv="${LOCK_DIR}/result_${_user}.tsv"
|
||||||
|
|
||||||
|
if $DRY_RUN ; then
|
||||||
|
echononl " Dry-run for account \033[1;37m${_user}\033[m (${_user_selected} to check).."
|
||||||
|
else
|
||||||
|
echononl " Deleting for account \033[1;37m${_user}\033[m (${_user_selected} to delete).."
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -s "$list_file" ]] ; then
|
||||||
|
if $DRY_RUN ; then
|
||||||
|
su -c "$PHP_BIN $delete_php_file $_user $list_file --dry-run" -s /bin/bash $HTTP_USER > "$user_result_tsv" 2> "$log_file"
|
||||||
|
else
|
||||||
|
su -c "$PHP_BIN $delete_php_file $_user $list_file" -s /bin/bash $HTTP_USER > "$user_result_tsv" 2> "$log_file"
|
||||||
|
fi
|
||||||
|
_rc=$?
|
||||||
|
else
|
||||||
|
echo -e "path\tstatus\tdetail" > "$user_result_tsv"
|
||||||
|
_rc=0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ $_rc -ne 0 ]]; then
|
||||||
|
echo_failed
|
||||||
|
error "$(cat "$log_file")"
|
||||||
|
echo " [ FEHLER beim Delete-Lauf - siehe Server-Log ]" >> "$delete_report_file"
|
||||||
|
(( total_failed_users++ ))
|
||||||
|
unset _user_selected
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo_done
|
||||||
|
$terminal && echo ""
|
||||||
|
|
||||||
|
declare -i _user_ok=0
|
||||||
|
declare -i _user_not_found=0
|
||||||
|
declare -i _user_errors=0
|
||||||
|
declare -a _user_error_lines=()
|
||||||
|
|
||||||
|
while IFS=$'\t' read -r _path _status _detail ; do
|
||||||
|
|
||||||
|
[[ "$_path" = "path" ]] && continue
|
||||||
|
[[ -z "$_path" ]] && continue
|
||||||
|
|
||||||
|
case "$_status" in
|
||||||
|
OK|DRY_RUN) (( _user_ok++ )) ;;
|
||||||
|
NOT_FOUND) (( _user_not_found++ )) ;;
|
||||||
|
*) (( _user_errors++ )); _user_error_lines+=("${_path}"$'\t'"${_status}: ${_detail}") ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
printf '%s\t%s\t%s\n' "$_path" "$_status" "$_detail" >> "$delete_report_file"
|
||||||
|
|
||||||
|
done < "$user_result_tsv"
|
||||||
|
|
||||||
|
_user_pct_ok="$(calc_percent "$_user_ok" "$_user_selected")"
|
||||||
|
|
||||||
|
if $terminal ; then
|
||||||
|
echo -e " \033[1;37mAccount ${_user}\033[m"
|
||||||
|
echo -e " Ausgewaehlt (aus der Pfadliste)............: ${_user_selected}"
|
||||||
|
if $DRY_RUN ; then
|
||||||
|
echo -e " Wuerde geloescht werden.....................: \033[1;32m${_user_ok} (${_user_pct_ok} %)\033[m"
|
||||||
|
else
|
||||||
|
if [[ $_user_ok -gt 0 ]] ; then
|
||||||
|
echo -e " Geloescht....................................: \033[1;32m${_user_ok} (${_user_pct_ok} %)\033[m"
|
||||||
|
else
|
||||||
|
echo -e " Geloescht....................................: ${_user_ok} (${_user_pct_ok} %)"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
if [[ $_user_not_found -gt 0 ]] ; then
|
||||||
|
echo -e " Bereits nicht mehr vorhanden................: ${_user_not_found}"
|
||||||
|
fi
|
||||||
|
if [[ $_user_errors -gt 0 ]] ; then
|
||||||
|
echo -e " Fehler beim Loeschen.........................: \033[1;31m${_user_errors}\033[m"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
fi
|
||||||
|
|
||||||
|
{
|
||||||
|
if [[ $_user_errors -gt 0 ]] ; then
|
||||||
|
echo ""
|
||||||
|
echo " ------------------------------------------------------------------"
|
||||||
|
echo " Fehler beim Loeschen - Account ${_user} (${_user_errors})"
|
||||||
|
echo " ------------------------------------------------------------------"
|
||||||
|
for _line in "${_user_error_lines[@]}" ; do
|
||||||
|
printf ' %s\n' "$_line"
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
} >> "$delete_report_file"
|
||||||
|
|
||||||
|
{
|
||||||
|
echo ""
|
||||||
|
echo " Account ${_user}"
|
||||||
|
echo " Ausgewaehlt (aus der Pfadliste)............: ${_user_selected}"
|
||||||
|
if $DRY_RUN ; then
|
||||||
|
echo " Wuerde geloescht werden.....................: ${_user_ok} (${_user_pct_ok} %)"
|
||||||
|
else
|
||||||
|
echo " Geloescht....................................: ${_user_ok} (${_user_pct_ok} %)"
|
||||||
|
fi
|
||||||
|
[[ $_user_not_found -gt 0 ]] && echo " Bereits nicht mehr vorhanden................: ${_user_not_found}"
|
||||||
|
[[ $_user_errors -gt 0 ]] && echo " Fehler beim Loeschen.........................: ${_user_errors}"
|
||||||
|
} >> "$delete_report_file"
|
||||||
|
|
||||||
|
(( total_ok += _user_ok ))
|
||||||
|
(( total_not_found += _user_not_found ))
|
||||||
|
(( total_errors += _user_errors ))
|
||||||
|
|
||||||
|
unset _user_selected _user_ok _user_not_found _user_errors _user_error_lines
|
||||||
|
|
||||||
|
done
|
||||||
|
|
||||||
|
|
||||||
|
total_pct_ok="$(calc_percent "$total_ok" "$total_selected")"
|
||||||
|
|
||||||
|
{
|
||||||
|
echo ""
|
||||||
|
echo ""
|
||||||
|
echo "=================================================================="
|
||||||
|
echo " Gesamtergebnis"
|
||||||
|
echo "=================================================================="
|
||||||
|
echo ""
|
||||||
|
echo "Betroffene Accounts..........................: ${#account_arr[@]}"
|
||||||
|
[[ $total_failed_users -gt 0 ]] && echo "Accounts mit Delete-Fehler...................: ${total_failed_users}"
|
||||||
|
echo "Ausgewaehlt (aus der Pfadliste)..............: ${total_selected}"
|
||||||
|
if $DRY_RUN ; then
|
||||||
|
echo "Wuerde geloescht werden insgesamt............: ${total_ok} (${total_pct_ok} %)"
|
||||||
|
else
|
||||||
|
echo "Geloescht insgesamt...........................: ${total_ok} (${total_pct_ok} %)"
|
||||||
|
fi
|
||||||
|
[[ $total_not_found -gt 0 ]] && echo "Bereits nicht mehr vorhanden insgesamt.......: ${total_not_found}"
|
||||||
|
[[ $total_errors -gt 0 ]] && echo "Fehler beim Loeschen insgesamt................: ${total_errors}"
|
||||||
|
echo ""
|
||||||
|
if $DRY_RUN ; then
|
||||||
|
echo "DRY-RUN: es wurde nichts gesichert oder geloescht."
|
||||||
|
else
|
||||||
|
echo "Chiffretext-Sicherung (vor dem Loeschen) liegt unter: ${backup_dir}"
|
||||||
|
fi
|
||||||
|
} >> "$delete_report_file"
|
||||||
|
|
||||||
|
blank_line
|
||||||
|
|
||||||
|
if $terminal ; then
|
||||||
|
echo -e "\033[37m\033[1mErgebnis\033[m"
|
||||||
|
echo ""
|
||||||
|
echo -e " Betroffene Accounts..........................: ${#account_arr[@]}"
|
||||||
|
[[ $total_failed_users -gt 0 ]] && echo -e " Accounts mit Delete-Fehler...................: \033[1;31m${total_failed_users}\033[m"
|
||||||
|
echo -e " Ausgewaehlt (aus der Pfadliste)..............: ${total_selected}"
|
||||||
|
if $DRY_RUN ; then
|
||||||
|
echo -e " Wuerde geloescht werden insgesamt............: \033[1;32m${total_ok} (${total_pct_ok} %)\033[m"
|
||||||
|
else
|
||||||
|
if [[ $total_ok -gt 0 ]] ; then
|
||||||
|
echo -e " Geloescht insgesamt...........................: \033[1;32m${total_ok} (${total_pct_ok} %)\033[m"
|
||||||
|
else
|
||||||
|
echo -e " Geloescht insgesamt...........................: ${total_ok} (${total_pct_ok} %)"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
[[ $total_not_found -gt 0 ]] && echo -e " Bereits nicht mehr vorhanden insgesamt.......: ${total_not_found}"
|
||||||
|
[[ $total_errors -gt 0 ]] && echo -e " Fehler beim Loeschen insgesamt................: \033[1;31m${total_errors}\033[m"
|
||||||
|
echo ""
|
||||||
|
echo -e " Delete-Report.................................: reports/$(basename "${delete_report_file}")"
|
||||||
|
if ! $DRY_RUN ; then
|
||||||
|
echo -e " Chiffretext-Sicherung.........................: $backup_dir"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
if $DRY_RUN ; then
|
||||||
|
info "Dry-run beendet - es wurde nichts geloescht. Ohne '-n' (oder mit [2] bei der Modusabfrage) fuer den echten Delete erneut ausfuehren."
|
||||||
|
else
|
||||||
|
warn "Bitte pruefen, ob die geloeschten Dateien wie erwartet im Trash der jeweiligen Accounts liegen (sofern 'files_trashbin' aktiviert ist)."
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
clean_up 0
|
||||||
+36
-9
@@ -48,17 +48,30 @@ usage() {
|
|||||||
fix - these are NOT signature problems, they are Server-Side
|
fix - these are NOT signature problems, they are Server-Side
|
||||||
Encryption KEY-MATERIAL problems (a per-user 'share key' for a file
|
Encryption KEY-MATERIAL problems (a per-user 'share key' for a file
|
||||||
is either missing on disk or fails to decrypt with that user's
|
is either missing on disk or fails to decrypt with that user's
|
||||||
private key). This script accepts either a recovery_*.tsv or a
|
private key).
|
||||||
restore_*.tsv report as input.
|
|
||||||
|
\033[1mThis script accepts either of two report types as input, and\033[m
|
||||||
|
\033[1mevaluates a different row/status from each:\033[m
|
||||||
|
|
||||||
|
\033[1mrecovery_*.tsv\033[m report (from recover_bad_signature.sh)
|
||||||
|
-> evaluates every row whose validation is \033[1mREAD_ERROR\033[m
|
||||||
|
|
||||||
|
\033[1mrestore_*.tsv\033[m report (from restore_bad_signature.sh)
|
||||||
|
-> evaluates every row whose status is \033[1mWRITE_ERROR\033[m
|
||||||
|
|
||||||
This script changes NOTHING. It never decrypts a file, never
|
This script changes NOTHING. It never decrypts a file, never
|
||||||
touches 'encryption_skip_signature_check', and never writes
|
touches 'encryption_skip_signature_check', and never writes
|
||||||
anywhere except its own report file. It only:
|
anywhere except its own report file. It only:
|
||||||
|
|
||||||
1. Reads a chosen recovery_*.tsv report (produced by
|
1. Reads the chosen report and picks out the matching rows:
|
||||||
recover_bad_signature.sh) and picks out every row whose
|
|
||||||
validation is READ_ERROR and whose detail text matches one of
|
- from a \033[1mrecovery_*.tsv\033[m report: every row whose
|
||||||
the two known key-material error signatures above.
|
validation is \033[1mREAD_ERROR\033[m
|
||||||
|
- from a \033[1mrestore_*.tsv\033[m report: every row whose
|
||||||
|
status is \033[1mWRITE_ERROR\033[m
|
||||||
|
|
||||||
|
...and, in both cases, whose detail text matches one of the
|
||||||
|
two known key-material error signatures above.
|
||||||
2. For each such file, resolves - via Nextcloud's normal,
|
2. For each such file, resolves - via Nextcloud's normal,
|
||||||
read-only Files API (\$node->getOwner(), \$folder->getById())
|
read-only Files API (\$node->getOwner(), \$folder->getById())
|
||||||
- who currently owns the file, i.e. whether the affected
|
- who currently owns the file, i.e. whether the affected
|
||||||
@@ -405,8 +418,11 @@ if $terminal ; then
|
|||||||
Nextcloud storage and never touches 'encryption_skip_signature_check'.
|
Nextcloud storage and never touches 'encryption_skip_signature_check'.
|
||||||
It only inspects, on disk and via the normal Files API, whether the
|
It only inspects, on disk and via the normal Files API, whether the
|
||||||
expected encryption key files are present for accounts/files that
|
expected encryption key files are present for accounts/files that
|
||||||
recover_bad_signature.sh could not read (READ_ERROR, not 'Bad
|
recover_bad_signature.sh or restore_bad_signature.sh could not
|
||||||
Signature').\033[m"
|
process (not 'Bad Signature').\033[m"
|
||||||
|
echo ""
|
||||||
|
echo -e " \033[1mrecovery_*.tsv\033[m report -> evaluates \033[1mREAD_ERROR\033[m rows"
|
||||||
|
echo -e " \033[1mrestore_*.tsv\033[m report -> evaluates \033[1mWRITE_ERROR\033[m rows"
|
||||||
echo -e "\033[32m-----\033[m"
|
echo -e "\033[32m-----\033[m"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
@@ -629,6 +645,13 @@ if ($oldWorkingDir === false) {
|
|||||||
}
|
}
|
||||||
chdir(__DIR__);
|
chdir(__DIR__);
|
||||||
require_once __DIR__ . '/lib/base.php';
|
require_once __DIR__ . '/lib/base.php';
|
||||||
|
// Nextcloud's own bootstrap (lib/base.php) unconditionally calls
|
||||||
|
// set_time_limit(3600) as part of the require above - this OVERRIDES
|
||||||
|
// whatever -d max_execution_time was passed on the PHP command line,
|
||||||
|
// since it is a later, explicit runtime call. Undo that here, now that
|
||||||
|
// the require is done, so diagnosing a large number of files isn't
|
||||||
|
// killed after exactly one hour regardless of the CLI flag.
|
||||||
|
set_time_limit(0);
|
||||||
chdir($oldWorkingDir);
|
chdir($oldWorkingDir);
|
||||||
|
|
||||||
if (function_exists('posix_getuid') && posix_getuid() === 0) {
|
if (function_exists('posix_getuid') && posix_getuid() === 0) {
|
||||||
@@ -775,7 +798,11 @@ for _user in "${selected_user_arr[@]}" ; do
|
|||||||
owner_result_tsv="${LOCK_DIR}/owner_${_user}.tsv"
|
owner_result_tsv="${LOCK_DIR}/owner_${_user}.tsv"
|
||||||
|
|
||||||
echononl " Resolving ownership for account \033[1;37m${_user}\033[m (${_user_selected} files).."
|
echononl " Resolving ownership for account \033[1;37m${_user}\033[m (${_user_selected} files).."
|
||||||
su -c "$PHP_BIN $diag_php_file $_user $path_list_file" -s /bin/bash $HTTP_USER > "$owner_result_tsv" 2> "$log_file"
|
# -d max_execution_time=0: resolving ownership/key-material for a
|
||||||
|
# large number of files can run well over an hour; without this the
|
||||||
|
# PHP CLI process is killed by PHP's own execution-time limit
|
||||||
|
# (typically inherited from the webserver's php.ini) mid-run.
|
||||||
|
su -c "$PHP_BIN -d max_execution_time=0 $diag_php_file $_user $path_list_file" -s /bin/bash $HTTP_USER > "$owner_result_tsv" 2> "$log_file"
|
||||||
_rc=$?
|
_rc=$?
|
||||||
|
|
||||||
if [[ $_rc -ne 0 ]]; then
|
if [[ $_rc -ne 0 ]]; then
|
||||||
|
|||||||
+284
-14
@@ -148,6 +148,8 @@ restore_encryption_flag() {
|
|||||||
clean_up() {
|
clean_up() {
|
||||||
|
|
||||||
# Perform program exit housekeeping
|
# Perform program exit housekeeping
|
||||||
|
# Clear EXIT trap first so that the exit below does not fire it again.
|
||||||
|
trap - EXIT
|
||||||
restore_encryption_flag
|
restore_encryption_flag
|
||||||
[[ -n "$recovery_php_file" ]] && rm -f "$recovery_php_file" 2> /dev/null
|
[[ -n "$recovery_php_file" ]] && rm -f "$recovery_php_file" 2> /dev/null
|
||||||
rm -rf "$LOCK_DIR"
|
rm -rf "$LOCK_DIR"
|
||||||
@@ -155,6 +157,82 @@ clean_up() {
|
|||||||
exit $1
|
exit $1
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
check_optional_validation_tools() {
|
||||||
|
|
||||||
|
# Checks whether optional tools that improve file-validation quality
|
||||||
|
# are installed. In interactive mode the user is offered to install
|
||||||
|
# any that are missing via apt. In non-interactive mode the function
|
||||||
|
# is a quiet no-op (missing tools degrade quality but do not abort).
|
||||||
|
|
||||||
|
local _missing_tools=()
|
||||||
|
local _missing_pkgs=()
|
||||||
|
local _desc=()
|
||||||
|
|
||||||
|
command -v identify > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("identify")
|
||||||
|
_missing_pkgs+=("imagemagick")
|
||||||
|
_desc+=("identify (imagemagick) — PNG / TIFF / BMP / GIF: vollst. Dekodierung statt nur Magic-Bytes")
|
||||||
|
}
|
||||||
|
command -v ffprobe > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("ffprobe")
|
||||||
|
_missing_pkgs+=("ffmpeg")
|
||||||
|
_desc+=("ffprobe (ffmpeg) — MP4 / MOV / M4V: Container-Parsing statt nur ftyp-Box-Suche")
|
||||||
|
}
|
||||||
|
command -v mp3val > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("mp3val")
|
||||||
|
_missing_pkgs+=("mp3val")
|
||||||
|
_desc+=("mp3val (mp3val) — MP3: Frame-Struktur-Check statt nur ID3-Signatur")
|
||||||
|
}
|
||||||
|
command -v flac > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("flac")
|
||||||
|
_missing_pkgs+=("flac")
|
||||||
|
_desc+=("flac (flac) — FLAC: Decode-Test statt nur fLaC-Signatur")
|
||||||
|
}
|
||||||
|
command -v ogginfo > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("ogginfo")
|
||||||
|
_missing_pkgs+=("vorbis-tools")
|
||||||
|
_desc+=("ogginfo (vorbis-tools) — OGG / OGA / OPUS: Container-Check statt nur OggS-Signatur")
|
||||||
|
}
|
||||||
|
|
||||||
|
[[ ${#_missing_tools[@]} -eq 0 ]] && return 0
|
||||||
|
|
||||||
|
if ! $terminal ; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
local _pkg_list="${_missing_pkgs[*]}"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e " \033[33mOptionale Validierungs-Tools fehlen – Checks laufen mit reduzierter Genauigkeit:\033[m"
|
||||||
|
echo ""
|
||||||
|
local _d
|
||||||
|
for _d in "${_desc[@]}" ; do
|
||||||
|
echo " ${_d}"
|
||||||
|
done
|
||||||
|
echo ""
|
||||||
|
echo -n " Jetzt installieren? apt install ${_pkg_list} [j/N]: "
|
||||||
|
read -r _yn
|
||||||
|
echo ""
|
||||||
|
if [[ "$_yn" =~ ^[jJyY]$ ]] ; then
|
||||||
|
echononl " Installiere Pakete: ${_pkg_list}.."
|
||||||
|
# shellcheck disable=SC2086
|
||||||
|
if apt-get install -y ${_missing_pkgs[*]} > /dev/null 2>&1 ; then
|
||||||
|
echo_ok
|
||||||
|
else
|
||||||
|
echo_failed
|
||||||
|
echo ""
|
||||||
|
echo " Bitte manuell installieren:"
|
||||||
|
echo " apt install ${_pkg_list}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo " Übersprungen — Validierung läuft mit reduzierter Genauigkeit."
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
is_number() {
|
is_number() {
|
||||||
|
|
||||||
return $(test ! -z "${1##*[!0-9]*}" > /dev/null 2>&1);
|
return $(test ! -z "${1##*[!0-9]*}" > /dev/null 2>&1);
|
||||||
@@ -303,38 +381,71 @@ validate_recovered_file() {
|
|||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
png)
|
png)
|
||||||
|
if command -v identify > /dev/null 2>&1 ; then
|
||||||
|
# Full decode: catches truncated data, corrupt IDAT chunks, bad CRCs
|
||||||
|
if identify -regard-warnings "$_f" > /dev/null 2>&1 ; then
|
||||||
|
echo "VALID|ImageMagick 'identify' decoded PNG ok"
|
||||||
|
else
|
||||||
|
echo "INVALID|ImageMagick 'identify' failed to decode PNG"
|
||||||
|
fi
|
||||||
|
else
|
||||||
local _head
|
local _head
|
||||||
_head="$(head -c8 "$_f" 2> /dev/null | od -An -tx1 | tr -d ' \n')"
|
_head="$(head -c8 "$_f" 2> /dev/null | od -An -tx1 | tr -d ' \n')"
|
||||||
if [[ "$_head" = "89504e470d0a1a0a" ]] ; then
|
if [[ "$_head" = "89504e470d0a1a0a" ]] ; then
|
||||||
echo "VALID|PNG signature ok"
|
echo "VALID|PNG signature ok (identify not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|PNG signature missing"
|
echo "INVALID|PNG signature missing"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
gif)
|
gif)
|
||||||
|
if command -v identify > /dev/null 2>&1 ; then
|
||||||
|
if identify -regard-warnings "$_f" > /dev/null 2>&1 ; then
|
||||||
|
echo "VALID|ImageMagick 'identify' decoded GIF ok"
|
||||||
|
else
|
||||||
|
echo "INVALID|ImageMagick 'identify' failed to decode GIF"
|
||||||
|
fi
|
||||||
|
else
|
||||||
local _head6
|
local _head6
|
||||||
_head6="$(head -c6 "$_f" 2> /dev/null)"
|
_head6="$(head -c6 "$_f" 2> /dev/null)"
|
||||||
if [[ "$_head6" = "GIF87a" || "$_head6" = "GIF89a" ]] ; then
|
if [[ "$_head6" = "GIF87a" || "$_head6" = "GIF89a" ]] ; then
|
||||||
echo "VALID|GIF signature ok"
|
echo "VALID|GIF signature ok (identify not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|GIF signature missing"
|
echo "INVALID|GIF signature missing"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
bmp)
|
bmp)
|
||||||
|
if command -v identify > /dev/null 2>&1 ; then
|
||||||
|
if identify -regard-warnings "$_f" > /dev/null 2>&1 ; then
|
||||||
|
echo "VALID|ImageMagick 'identify' decoded BMP ok"
|
||||||
|
else
|
||||||
|
echo "INVALID|ImageMagick 'identify' failed to decode BMP"
|
||||||
|
fi
|
||||||
|
else
|
||||||
if [[ "$(head -c2 "$_f" 2> /dev/null)" = "BM" ]] ; then
|
if [[ "$(head -c2 "$_f" 2> /dev/null)" = "BM" ]] ; then
|
||||||
echo "VALID|BMP signature ok"
|
echo "VALID|BMP signature ok (identify not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|BMP signature missing"
|
echo "INVALID|BMP signature missing"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
tif|tiff)
|
tif|tiff)
|
||||||
|
if command -v identify > /dev/null 2>&1 ; then
|
||||||
|
if identify -regard-warnings "$_f" > /dev/null 2>&1 ; then
|
||||||
|
echo "VALID|ImageMagick 'identify' decoded TIFF ok"
|
||||||
|
else
|
||||||
|
echo "INVALID|ImageMagick 'identify' failed to decode TIFF"
|
||||||
|
fi
|
||||||
|
else
|
||||||
local _head4
|
local _head4
|
||||||
_head4="$(head -c4 "$_f" 2> /dev/null | od -An -tx1 | tr -d ' \n')"
|
_head4="$(head -c4 "$_f" 2> /dev/null | od -An -tx1 | tr -d ' \n')"
|
||||||
if [[ "$_head4" = "49492a00" || "$_head4" = "4d4d002a" ]] ; then
|
if [[ "$_head4" = "49492a00" || "$_head4" = "4d4d002a" ]] ; then
|
||||||
echo "VALID|TIFF signature ok"
|
echo "VALID|TIFF signature ok (identify not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|TIFF signature missing"
|
echo "INVALID|TIFF signature missing"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
pnm|pgm|ppm|pbm)
|
pnm|pgm|ppm|pbm)
|
||||||
local _head2
|
local _head2
|
||||||
@@ -360,23 +471,59 @@ validate_recovered_file() {
|
|||||||
if [[ "$_head8o" = "d0cf11e0a1b11ae1" ]] ; then
|
if [[ "$_head8o" = "d0cf11e0a1b11ae1" ]] ; then
|
||||||
echo "VALID|OLE2/CFBF container signature ok - this is a password-protected Office file (encrypted package), not a plain zip, so the zip check does not apply; open it with the password to verify content"
|
echo "VALID|OLE2/CFBF container signature ok - this is a password-protected Office file (encrypted package), not a plain zip, so the zip check does not apply; open it with the password to verify content"
|
||||||
elif command -v unzip > /dev/null 2>&1 ; then
|
elif command -v unzip > /dev/null 2>&1 ; then
|
||||||
if unzip -tq "$_f" > /dev/null 2>&1 ; then
|
# Run unzip inside setsid so it has no controlling terminal.
|
||||||
|
# Without setsid, a password-protected ZIP causes unzip to try
|
||||||
|
# opening /dev/tty to prompt for the password. When running in
|
||||||
|
# the background of a tmux session this generates SIGTTIN, which
|
||||||
|
# STOPS the process (ps state T). A stopped process cannot receive
|
||||||
|
# SIGTERM, so timeout hangs indefinitely waiting for a child that
|
||||||
|
# will never exit. With setsid the /dev/tty open fails immediately
|
||||||
|
# with ENXIO and unzip exits with a non-zero code instead.
|
||||||
|
# -k 5: send SIGKILL 5 s after SIGTERM in case the process is
|
||||||
|
# still alive (e.g. stopped or in uninterruptible sleep).
|
||||||
|
local _unzip_exit
|
||||||
|
timeout -k 5 120 setsid unzip -tq "$_f" < /dev/null > /dev/null 2>&1
|
||||||
|
_unzip_exit=$?
|
||||||
|
if [[ $_unzip_exit -eq 0 ]] ; then
|
||||||
echo "VALID|zip integrity ok"
|
echo "VALID|zip integrity ok"
|
||||||
|
elif [[ $_unzip_exit -eq 124 || $_unzip_exit -eq 137 ]] ; then
|
||||||
|
# 124 = killed by SIGTERM after timeout, 137 = killed by SIGKILL (128+9)
|
||||||
|
echo "UNVERIFIED|zip integrity check timed out after 120 s (file may be very large or corrupt; check manually with: unzip -t \"$_f\")"
|
||||||
else
|
else
|
||||||
local _badentry
|
local _badentry
|
||||||
_badentry="$(trim "$(unzip -t "$_f" 2>&1 | grep -v '^Archive:' | grep -v '^[[:space:]]*$' | head -1)")"
|
_badentry="$(trim "$(timeout -k 5 120 setsid unzip -t "$_f" < /dev/null 2>&1 | grep -v '^Archive:' | grep -v '^[[:space:]]*$' | head -1)")"
|
||||||
|
if echo "$_badentry" | grep -qi "password\|encrypt\|need PK compat" ; then
|
||||||
|
echo "UNVERIFIED|zip is password-protected (cannot verify without password${_badentry:+; unzip says: ${_badentry}})"
|
||||||
|
else
|
||||||
echo "INVALID|zip integrity check failed${_badentry:+ (${_badentry})}"
|
echo "INVALID|zip integrity check failed${_badentry:+ (${_badentry})}"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
else
|
else
|
||||||
echo "UNVERIFIED|unzip not installed"
|
echo "UNVERIFIED|unzip not installed"
|
||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
mp4|mov|m4v)
|
mp4|mov|m4v)
|
||||||
|
if command -v ffprobe > /dev/null 2>&1 ; then
|
||||||
|
# Full container parse: detects truncated/corrupt streams
|
||||||
|
local _ffprobe_out _ffprobe_exit
|
||||||
|
_ffprobe_out="$(ffprobe -v error -show_streams "$_f" 2>&1)"
|
||||||
|
_ffprobe_exit=$?
|
||||||
|
if [[ $_ffprobe_exit -eq 0 ]] ; then
|
||||||
|
local _streams
|
||||||
|
_streams="$(echo "$_ffprobe_out" | grep -c '\[STREAM\]' || true)"
|
||||||
|
echo "VALID|ffprobe parsed container ok (${_streams} stream(s) found)"
|
||||||
|
else
|
||||||
|
local _ffprobe_err
|
||||||
|
_ffprobe_err="$(echo "$_ffprobe_out" | head -1)"
|
||||||
|
echo "INVALID|ffprobe failed to parse container${_ffprobe_err:+ (${_ffprobe_err})}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
if head -c 64 "$_f" 2> /dev/null | grep -aq "ftyp" ; then
|
if head -c 64 "$_f" 2> /dev/null | grep -aq "ftyp" ; then
|
||||||
echo "VALID|mp4 ftyp box found"
|
echo "VALID|mp4 ftyp box found (ffprobe not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|mp4 ftyp box not found"
|
echo "INVALID|mp4 ftyp box not found"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
doc|xls|ppt|ole|msi)
|
doc|xls|ppt|ole|msi)
|
||||||
local _head8
|
local _head8
|
||||||
@@ -475,6 +622,75 @@ validate_recovered_file() {
|
|||||||
echo "INVALID|GIMP (gimp xcf) signature missing"
|
echo "INVALID|GIMP (gimp xcf) signature missing"
|
||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
|
mp3)
|
||||||
|
if command -v mp3val > /dev/null 2>&1 ; then
|
||||||
|
# mp3val always exits 0 but prints "No errors found" on success
|
||||||
|
local _mp3val_out
|
||||||
|
_mp3val_out="$(mp3val "$_f" 2>&1)"
|
||||||
|
if echo "$_mp3val_out" | grep -q 'No errors found' ; then
|
||||||
|
echo "VALID|mp3val: no errors found"
|
||||||
|
else
|
||||||
|
local _mp3_err
|
||||||
|
_mp3_err="$(echo "$_mp3val_out" | grep -iv '^mp3val\|^$' | head -2 | tr '\n' ' ' | sed 's/ $//')"
|
||||||
|
echo "INVALID|mp3val reported errors${_mp3_err:+ (${_mp3_err})}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
local _head3 _head2hex
|
||||||
|
_head3="$(head -c3 "$_f" 2>/dev/null)"
|
||||||
|
_head2hex="$(head -c2 "$_f" 2>/dev/null | od -An -tx1 | tr -d ' \n')"
|
||||||
|
if [[ "$_head3" = "ID3" ]] || echo "$_head2hex" | grep -qE '^fff[bef2]' ; then
|
||||||
|
echo "VALID|MP3 ID3 tag / MPEG sync ok (mp3val not installed — no deep check)"
|
||||||
|
else
|
||||||
|
echo "INVALID|MP3: no ID3 header or MPEG frame sync found"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
flac)
|
||||||
|
if command -v flac > /dev/null 2>&1 ; then
|
||||||
|
# --test decodes without writing output; exit 0 = file is intact
|
||||||
|
local _flac_out _flac_exit
|
||||||
|
_flac_out="$(flac --silent --test "$_f" 2>&1)"
|
||||||
|
_flac_exit=$?
|
||||||
|
if [[ $_flac_exit -eq 0 ]] ; then
|
||||||
|
echo "VALID|flac --test: ok"
|
||||||
|
else
|
||||||
|
local _flac_err
|
||||||
|
_flac_err="$(echo "$_flac_out" | grep -v '^$' | tail -1)"
|
||||||
|
echo "INVALID|flac --test failed${_flac_err:+ (${_flac_err})}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
local _head4hex
|
||||||
|
_head4hex="$(head -c4 "$_f" 2>/dev/null | od -An -tx1 | tr -d ' \n')"
|
||||||
|
if [[ "$_head4hex" = "664c6143" ]] ; then # "fLaC"
|
||||||
|
echo "VALID|FLAC 'fLaC' signature ok (flac not installed — no deep check)"
|
||||||
|
else
|
||||||
|
echo "INVALID|FLAC 'fLaC' signature missing"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
ogg|oga|ogv|opus)
|
||||||
|
if command -v ogginfo > /dev/null 2>&1 ; then
|
||||||
|
# ogginfo exits 0 if the Ogg container is intact
|
||||||
|
local _ogg_out _ogg_exit
|
||||||
|
_ogg_out="$(ogginfo "$_f" 2>&1)"
|
||||||
|
_ogg_exit=$?
|
||||||
|
if [[ $_ogg_exit -eq 0 ]] ; then
|
||||||
|
echo "VALID|ogginfo: container parsed ok"
|
||||||
|
else
|
||||||
|
local _ogg_err
|
||||||
|
_ogg_err="$(echo "$_ogg_out" | grep -iv '^Processing\|^$' | head -2 | tr '\n' ' ' | sed 's/ $//')"
|
||||||
|
echo "INVALID|ogginfo failed${_ogg_err:+ (${_ogg_err})}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
local _head4ogg
|
||||||
|
_head4ogg="$(head -c4 "$_f" 2>/dev/null)"
|
||||||
|
if [[ "$_head4ogg" = "OggS" ]] ; then
|
||||||
|
echo "VALID|OGG 'OggS' signature ok (ogginfo not installed — no deep check)"
|
||||||
|
else
|
||||||
|
echo "INVALID|OGG 'OggS' signature missing"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
;;
|
||||||
wav)
|
wav)
|
||||||
local _riff _wave
|
local _riff _wave
|
||||||
_riff="$(head -c4 "$_f" 2> /dev/null)"
|
_riff="$(head -c4 "$_f" 2> /dev/null)"
|
||||||
@@ -562,6 +778,15 @@ validate_recovered_file() {
|
|||||||
## - PDF structural check shared by the 'pdf' case and the 'ai'
|
## - PDF structural check shared by the 'pdf' case and the 'ai'
|
||||||
## - (PDF-compatible) case. Sets $_pdf_check_detail, returns 0/1.
|
## - (PDF-compatible) case. Sets $_pdf_check_detail, returns 0/1.
|
||||||
## -
|
## -
|
||||||
|
## - qpdf exit codes:
|
||||||
|
## - 0 no problems
|
||||||
|
## - 2 structural errors → file is genuinely corrupt/unreadable
|
||||||
|
## - 3 warnings only → minor spec non-conformances; every real
|
||||||
|
## - viewer opens the file without issues
|
||||||
|
## - Only exit code 2 is treated as INVALID here. Exit code 3 (warnings)
|
||||||
|
## - is reported as VALID with a note, preventing false positives for
|
||||||
|
## - real-world PDFs that have trivial non-conformances.
|
||||||
|
## -
|
||||||
_pdf_check() {
|
_pdf_check() {
|
||||||
local _f="$1"
|
local _f="$1"
|
||||||
if [[ "$(head -c5 "$_f" 2> /dev/null)" != "%PDF-" ]] ; then
|
if [[ "$(head -c5 "$_f" 2> /dev/null)" != "%PDF-" ]] ; then
|
||||||
@@ -569,11 +794,22 @@ _pdf_check() {
|
|||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
if command -v qpdf > /dev/null 2>&1 ; then
|
if command -v qpdf > /dev/null 2>&1 ; then
|
||||||
if qpdf --check "$_f" > /dev/null 2>&1 ; then
|
local _qpdf_out _qpdf_exit
|
||||||
|
_qpdf_out="$(qpdf --check "$_f" 2>&1)"
|
||||||
|
_qpdf_exit=$?
|
||||||
|
if [[ $_qpdf_exit -eq 0 ]] ; then
|
||||||
_pdf_check_detail="qpdf --check ok"
|
_pdf_check_detail="qpdf --check ok"
|
||||||
return 0
|
return 0
|
||||||
|
elif [[ $_qpdf_exit -eq 3 ]] ; then
|
||||||
|
# Warnings only — no structural errors. File is readable by all
|
||||||
|
# standard PDF viewers; non-conformances are minor/cosmetic.
|
||||||
|
_pdf_check_detail="qpdf --check ok (warnings only — file is readable)"
|
||||||
|
return 0
|
||||||
else
|
else
|
||||||
_pdf_check_detail="qpdf --check failed"
|
# Exit code 2 (or unexpected): genuine structural errors.
|
||||||
|
local _first_err
|
||||||
|
_first_err="$(echo "$_qpdf_out" | grep -i 'error' | head -1 | sed 's/^[[:space:]]*//')"
|
||||||
|
_pdf_check_detail="qpdf --check failed (exit ${_qpdf_exit}${_first_err:+: ${_first_err}})"
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
@@ -764,7 +1000,7 @@ if mkdir "$LOCK_DIR" 2> /dev/null ; then
|
|||||||
|
|
||||||
# - Remove lockdir when the script finishes, or when it receives a signal
|
# - Remove lockdir when the script finishes, or when it receives a signal
|
||||||
# -
|
# -
|
||||||
trap clean_up SIGHUP SIGINT SIGTERM
|
trap 'clean_up 1' SIGHUP SIGINT SIGTERM
|
||||||
|
|
||||||
else
|
else
|
||||||
|
|
||||||
@@ -813,8 +1049,14 @@ if ! $_revalidate_only_explicit && $terminal ; then
|
|||||||
blank_line
|
blank_line
|
||||||
echo -e "\033[37m\033[1mWhich mode should this run use?\033[m"
|
echo -e "\033[37m\033[1mWhich mode should this run use?\033[m"
|
||||||
echo ""
|
echo ""
|
||||||
echo -e " \033[1m[1] Recovery\033[m - decrypt bad-signature files (temporarily skips the signature check), write them under ${DEFAULT_RECOVERY_BASE_DIR}/<website> and validate them"
|
echo -e " \033[1m[1] Recovery\033[m - decrypt bad-signature files (temporarily skips the signature check),
|
||||||
echo " [2] Revalidate-only - re-run just the validation checks against files a previous recovery run already wrote to disk (same as '-V'); nothing is decrypted again and no config value is touched"
|
write them under
|
||||||
|
\033[1m${DEFAULT_RECOVERY_BASE_DIR}/<website>\033[m
|
||||||
|
and validate them"
|
||||||
|
echo ""
|
||||||
|
echo " [2] Revalidate-only - re-run just the validation checks against files a previous recovery run
|
||||||
|
already wrote to disk (same as '-V'); nothing is decrypted again and
|
||||||
|
no config value is touched"
|
||||||
info "Just press Return to use the default: [1] Recovery."
|
info "Just press Return to use the default: [1] Recovery."
|
||||||
echo -n " Select mode by number [1]: "
|
echo -n " Select mode by number [1]: "
|
||||||
read _mode_choice
|
read _mode_choice
|
||||||
@@ -1077,6 +1319,8 @@ done
|
|||||||
mkdir -p "$report_dir" 2> /dev/null
|
mkdir -p "$report_dir" 2> /dev/null
|
||||||
recovery_report_file="${report_dir}/recovery_${WEBSITE}_${run_date}.tsv"
|
recovery_report_file="${report_dir}/recovery_${WEBSITE}_${run_date}.tsv"
|
||||||
|
|
||||||
|
check_optional_validation_tools
|
||||||
|
|
||||||
if $terminal ; then
|
if $terminal ; then
|
||||||
echo ""
|
echo ""
|
||||||
if $REVALIDATE_ONLY ; then
|
if $REVALIDATE_ONLY ; then
|
||||||
@@ -1117,6 +1361,18 @@ if $terminal ; then
|
|||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
if $terminal ; then
|
||||||
|
echo ""
|
||||||
|
echo -e " \033[1mTemporäre Dateien – werden laufend aktualisiert:\033[m"
|
||||||
|
echo ""
|
||||||
|
echo -e " Fortschritt / Debug-Meldungen (PHP-Ausgabe auf STDERR):"
|
||||||
|
echo -e " \033[1mtail -f ${log_file}\033[m"
|
||||||
|
echo ""
|
||||||
|
echo -e " Rohergebnis pro Account (PHP-Ausgabe auf STDOUT, Datei für Datei):"
|
||||||
|
echo -e " \033[1mtail -f ${LOCK_DIR}/result_<account>.tsv\033[m"
|
||||||
|
echo ""
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
||||||
{
|
{
|
||||||
echo "=================================================================="
|
echo "=================================================================="
|
||||||
@@ -1147,7 +1403,10 @@ su -c "$PHP_BIN $INSTALL_DIR/occ config:system:set encryption_skip_signature_che
|
|||||||
if [[ $? -eq 0 ]] ; then
|
if [[ $? -eq 0 ]] ; then
|
||||||
echo_ok
|
echo_ok
|
||||||
_encryption_flag_changed=true
|
_encryption_flag_changed=true
|
||||||
trap 'restore_encryption_flag; clean_up 1' SIGHUP SIGINT SIGTERM
|
# Trap signals AND normal/abnormal exit so that the encryption flag
|
||||||
|
# and temp files are always cleaned up — even on a syntax error or
|
||||||
|
# an unexpected crash (EXIT fires for any bash exit, including errors).
|
||||||
|
trap 'clean_up 1' SIGHUP SIGINT SIGTERM EXIT
|
||||||
else
|
else
|
||||||
echo_failed
|
echo_failed
|
||||||
fatal "Could not enable encryption_skip_signature_check: $(cat "$log_file")"
|
fatal "Could not enable encryption_skip_signature_check: $(cat "$log_file")"
|
||||||
@@ -1199,6 +1458,13 @@ if ($oldWorkingDir === false) {
|
|||||||
}
|
}
|
||||||
chdir(__DIR__);
|
chdir(__DIR__);
|
||||||
require_once __DIR__ . '/lib/base.php';
|
require_once __DIR__ . '/lib/base.php';
|
||||||
|
// Nextcloud's own bootstrap (lib/base.php) unconditionally calls
|
||||||
|
// set_time_limit(3600) as part of the require above - this OVERRIDES
|
||||||
|
// whatever -d max_execution_time was passed on the PHP command line,
|
||||||
|
// since it is a later, explicit runtime call. Undo that here, now that
|
||||||
|
// the require is done, so recovering a large account isn't killed
|
||||||
|
// after exactly one hour regardless of the CLI flag.
|
||||||
|
set_time_limit(0);
|
||||||
chdir($oldWorkingDir);
|
chdir($oldWorkingDir);
|
||||||
|
|
||||||
if (function_exists('posix_getuid') && posix_getuid() === 0) {
|
if (function_exists('posix_getuid') && posix_getuid() === 0) {
|
||||||
@@ -1433,7 +1699,11 @@ for _user in "${selected_user_arr[@]}" ; do
|
|||||||
chown -R "$HTTP_USER":"$HTTP_GROUP" "$recovery_dir" 2> /dev/null
|
chown -R "$HTTP_USER":"$HTTP_GROUP" "$recovery_dir" 2> /dev/null
|
||||||
|
|
||||||
echononl " Recovering account \033[1;37m${_user}\033[m (${_user_total} files).."
|
echononl " Recovering account \033[1;37m${_user}\033[m (${_user_total} files).."
|
||||||
su -c "$PHP_BIN $recovery_php_file $_user $list_file $user_out_dir" -s /bin/bash $HTTP_USER > "$user_result_tsv" 2> "$log_file"
|
# -d max_execution_time=0: decrypting/validating every file of a
|
||||||
|
# large account can run well over an hour; without this the PHP
|
||||||
|
# CLI process is killed by PHP's own execution-time limit
|
||||||
|
# (typically inherited from the webserver's php.ini) mid-run.
|
||||||
|
su -c "$PHP_BIN -d max_execution_time=0 $recovery_php_file $_user $list_file $user_out_dir" -s /bin/bash $HTTP_USER > "$user_result_tsv" 2> "$log_file"
|
||||||
_rc=$?
|
_rc=$?
|
||||||
|
|
||||||
fi
|
fi
|
||||||
|
|||||||
+269
-13
@@ -183,6 +183,8 @@ usage() {
|
|||||||
clean_up() {
|
clean_up() {
|
||||||
|
|
||||||
# Perform program exit housekeeping
|
# Perform program exit housekeeping
|
||||||
|
# Clear EXIT trap first so that the exit below does not fire it again.
|
||||||
|
trap - EXIT
|
||||||
[[ -n "$recreate_php_file" ]] && rm -f "$recreate_php_file" 2> /dev/null
|
[[ -n "$recreate_php_file" ]] && rm -f "$recreate_php_file" 2> /dev/null
|
||||||
rm -rf "$LOCK_DIR"
|
rm -rf "$LOCK_DIR"
|
||||||
blank_line
|
blank_line
|
||||||
@@ -190,6 +192,77 @@ clean_up() {
|
|||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
check_optional_validation_tools() {
|
||||||
|
|
||||||
|
local _missing_tools=()
|
||||||
|
local _missing_pkgs=()
|
||||||
|
local _desc=()
|
||||||
|
|
||||||
|
command -v identify > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("identify")
|
||||||
|
_missing_pkgs+=("imagemagick")
|
||||||
|
_desc+=("identify (imagemagick) — PNG / TIFF / BMP / GIF: vollst. Dekodierung statt nur Magic-Bytes")
|
||||||
|
}
|
||||||
|
command -v ffprobe > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("ffprobe")
|
||||||
|
_missing_pkgs+=("ffmpeg")
|
||||||
|
_desc+=("ffprobe (ffmpeg) — MP4 / MOV / M4V: Container-Parsing statt nur ftyp-Box-Suche")
|
||||||
|
}
|
||||||
|
command -v mp3val > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("mp3val")
|
||||||
|
_missing_pkgs+=("mp3val")
|
||||||
|
_desc+=("mp3val (mp3val) — MP3: Frame-Struktur-Check statt nur ID3-Signatur")
|
||||||
|
}
|
||||||
|
command -v flac > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("flac")
|
||||||
|
_missing_pkgs+=("flac")
|
||||||
|
_desc+=("flac (flac) — FLAC: Decode-Test statt nur fLaC-Signatur")
|
||||||
|
}
|
||||||
|
command -v ogginfo > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("ogginfo")
|
||||||
|
_missing_pkgs+=("vorbis-tools")
|
||||||
|
_desc+=("ogginfo (vorbis-tools) — OGG / OGA / OPUS: Container-Check statt nur OggS-Signatur")
|
||||||
|
}
|
||||||
|
|
||||||
|
[[ ${#_missing_tools[@]} -eq 0 ]] && return 0
|
||||||
|
|
||||||
|
if ! $terminal ; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
local _pkg_list="${_missing_pkgs[*]}"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e " \033[33mOptionale Validierungs-Tools fehlen – Checks laufen mit reduzierter Genauigkeit:\033[m"
|
||||||
|
echo ""
|
||||||
|
local _d
|
||||||
|
for _d in "${_desc[@]}" ; do
|
||||||
|
echo " ${_d}"
|
||||||
|
done
|
||||||
|
echo ""
|
||||||
|
echo -n " Jetzt installieren? apt install ${_pkg_list} [j/N]: "
|
||||||
|
read -r _yn
|
||||||
|
echo ""
|
||||||
|
if [[ "$_yn" =~ ^[jJyY]$ ]] ; then
|
||||||
|
echononl " Installiere Pakete: ${_pkg_list}.."
|
||||||
|
# shellcheck disable=SC2086
|
||||||
|
if apt-get install -y ${_missing_pkgs[*]} > /dev/null 2>&1 ; then
|
||||||
|
echo_ok
|
||||||
|
else
|
||||||
|
echo_failed
|
||||||
|
echo ""
|
||||||
|
echo " Bitte manuell installieren:"
|
||||||
|
echo " apt install ${_pkg_list}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo " Übersprungen — Validierung läuft mit reduzierter Genauigkeit."
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
is_number() {
|
is_number() {
|
||||||
|
|
||||||
return $(test ! -z "${1##*[!0-9]*}" > /dev/null 2>&1);
|
return $(test ! -z "${1##*[!0-9]*}" > /dev/null 2>&1);
|
||||||
@@ -338,38 +411,70 @@ validate_recovered_file() {
|
|||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
png)
|
png)
|
||||||
|
if command -v identify > /dev/null 2>&1 ; then
|
||||||
|
if identify -regard-warnings "$_f" > /dev/null 2>&1 ; then
|
||||||
|
echo "VALID|ImageMagick 'identify' decoded PNG ok"
|
||||||
|
else
|
||||||
|
echo "INVALID|ImageMagick 'identify' failed to decode PNG"
|
||||||
|
fi
|
||||||
|
else
|
||||||
local _head
|
local _head
|
||||||
_head="$(head -c8 "$_f" 2> /dev/null | od -An -tx1 | tr -d ' \n')"
|
_head="$(head -c8 "$_f" 2> /dev/null | od -An -tx1 | tr -d ' \n')"
|
||||||
if [[ "$_head" = "89504e470d0a1a0a" ]] ; then
|
if [[ "$_head" = "89504e470d0a1a0a" ]] ; then
|
||||||
echo "VALID|PNG signature ok"
|
echo "VALID|PNG signature ok (identify not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|PNG signature missing"
|
echo "INVALID|PNG signature missing"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
gif)
|
gif)
|
||||||
|
if command -v identify > /dev/null 2>&1 ; then
|
||||||
|
if identify -regard-warnings "$_f" > /dev/null 2>&1 ; then
|
||||||
|
echo "VALID|ImageMagick 'identify' decoded GIF ok"
|
||||||
|
else
|
||||||
|
echo "INVALID|ImageMagick 'identify' failed to decode GIF"
|
||||||
|
fi
|
||||||
|
else
|
||||||
local _head6
|
local _head6
|
||||||
_head6="$(head -c6 "$_f" 2> /dev/null)"
|
_head6="$(head -c6 "$_f" 2> /dev/null)"
|
||||||
if [[ "$_head6" = "GIF87a" || "$_head6" = "GIF89a" ]] ; then
|
if [[ "$_head6" = "GIF87a" || "$_head6" = "GIF89a" ]] ; then
|
||||||
echo "VALID|GIF signature ok"
|
echo "VALID|GIF signature ok (identify not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|GIF signature missing"
|
echo "INVALID|GIF signature missing"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
bmp)
|
bmp)
|
||||||
|
if command -v identify > /dev/null 2>&1 ; then
|
||||||
|
if identify -regard-warnings "$_f" > /dev/null 2>&1 ; then
|
||||||
|
echo "VALID|ImageMagick 'identify' decoded BMP ok"
|
||||||
|
else
|
||||||
|
echo "INVALID|ImageMagick 'identify' failed to decode BMP"
|
||||||
|
fi
|
||||||
|
else
|
||||||
if [[ "$(head -c2 "$_f" 2> /dev/null)" = "BM" ]] ; then
|
if [[ "$(head -c2 "$_f" 2> /dev/null)" = "BM" ]] ; then
|
||||||
echo "VALID|BMP signature ok"
|
echo "VALID|BMP signature ok (identify not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|BMP signature missing"
|
echo "INVALID|BMP signature missing"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
tif|tiff)
|
tif|tiff)
|
||||||
|
if command -v identify > /dev/null 2>&1 ; then
|
||||||
|
if identify -regard-warnings "$_f" > /dev/null 2>&1 ; then
|
||||||
|
echo "VALID|ImageMagick 'identify' decoded TIFF ok"
|
||||||
|
else
|
||||||
|
echo "INVALID|ImageMagick 'identify' failed to decode TIFF"
|
||||||
|
fi
|
||||||
|
else
|
||||||
local _head4
|
local _head4
|
||||||
_head4="$(head -c4 "$_f" 2> /dev/null | od -An -tx1 | tr -d ' \n')"
|
_head4="$(head -c4 "$_f" 2> /dev/null | od -An -tx1 | tr -d ' \n')"
|
||||||
if [[ "$_head4" = "49492a00" || "$_head4" = "4d4d002a" ]] ; then
|
if [[ "$_head4" = "49492a00" || "$_head4" = "4d4d002a" ]] ; then
|
||||||
echo "VALID|TIFF signature ok"
|
echo "VALID|TIFF signature ok (identify not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|TIFF signature missing"
|
echo "INVALID|TIFF signature missing"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
pnm|pgm|ppm|pbm)
|
pnm|pgm|ppm|pbm)
|
||||||
local _head2
|
local _head2
|
||||||
@@ -386,23 +491,58 @@ validate_recovered_file() {
|
|||||||
if [[ "$_head8o" = "d0cf11e0a1b11ae1" ]] ; then
|
if [[ "$_head8o" = "d0cf11e0a1b11ae1" ]] ; then
|
||||||
echo "VALID|OLE2/CFBF container signature ok - this is a password-protected Office file (encrypted package), not a plain zip, so the zip check does not apply; open it with the password to verify content"
|
echo "VALID|OLE2/CFBF container signature ok - this is a password-protected Office file (encrypted package), not a plain zip, so the zip check does not apply; open it with the password to verify content"
|
||||||
elif command -v unzip > /dev/null 2>&1 ; then
|
elif command -v unzip > /dev/null 2>&1 ; then
|
||||||
if unzip -tq "$_f" > /dev/null 2>&1 ; then
|
# Run unzip inside setsid so it has no controlling terminal.
|
||||||
|
# Without setsid, a password-protected ZIP causes unzip to try
|
||||||
|
# opening /dev/tty to prompt for the password. When running in
|
||||||
|
# the background of a tmux session this generates SIGTTIN, which
|
||||||
|
# STOPS the process (ps state T). A stopped process cannot receive
|
||||||
|
# SIGTERM, so timeout hangs indefinitely waiting for a child that
|
||||||
|
# will never exit. With setsid the /dev/tty open fails immediately
|
||||||
|
# with ENXIO and unzip exits with a non-zero code instead.
|
||||||
|
# -k 5: send SIGKILL 5 s after SIGTERM in case the process is
|
||||||
|
# still alive (e.g. stopped or in uninterruptible sleep).
|
||||||
|
local _unzip_exit
|
||||||
|
timeout -k 5 120 setsid unzip -tq "$_f" < /dev/null > /dev/null 2>&1
|
||||||
|
_unzip_exit=$?
|
||||||
|
if [[ $_unzip_exit -eq 0 ]] ; then
|
||||||
echo "VALID|zip integrity ok"
|
echo "VALID|zip integrity ok"
|
||||||
|
elif [[ $_unzip_exit -eq 124 || $_unzip_exit -eq 137 ]] ; then
|
||||||
|
# 124 = killed by SIGTERM after timeout, 137 = killed by SIGKILL (128+9)
|
||||||
|
echo "UNVERIFIED|zip integrity check timed out after 120 s (file may be very large or corrupt; check manually with: unzip -t \"$_f\")"
|
||||||
else
|
else
|
||||||
local _badentry
|
local _badentry
|
||||||
_badentry="$(trim "$(unzip -t "$_f" 2>&1 | grep -v '^Archive:' | grep -v '^[[:space:]]*$' | head -1)")"
|
_badentry="$(trim "$(timeout -k 5 120 setsid unzip -t "$_f" < /dev/null 2>&1 | grep -v '^Archive:' | grep -v '^[[:space:]]*$' | head -1)")"
|
||||||
|
if echo "$_badentry" | grep -qi "password\|encrypt\|need PK compat" ; then
|
||||||
|
echo "UNVERIFIED|zip is password-protected (cannot verify without password${_badentry:+; unzip says: ${_badentry}})"
|
||||||
|
else
|
||||||
echo "INVALID|zip integrity check failed${_badentry:+ (${_badentry})}"
|
echo "INVALID|zip integrity check failed${_badentry:+ (${_badentry})}"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
else
|
else
|
||||||
echo "UNVERIFIED|unzip not installed"
|
echo "UNVERIFIED|unzip not installed"
|
||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
mp4|mov|m4v)
|
mp4|mov|m4v)
|
||||||
|
if command -v ffprobe > /dev/null 2>&1 ; then
|
||||||
|
local _ffprobe_out _ffprobe_exit
|
||||||
|
_ffprobe_out="$(ffprobe -v error -show_streams "$_f" 2>&1)"
|
||||||
|
_ffprobe_exit=$?
|
||||||
|
if [[ $_ffprobe_exit -eq 0 ]] ; then
|
||||||
|
local _streams
|
||||||
|
_streams="$(echo "$_ffprobe_out" | grep -c '\[STREAM\]' || true)"
|
||||||
|
echo "VALID|ffprobe parsed container ok (${_streams} stream(s) found)"
|
||||||
|
else
|
||||||
|
local _ffprobe_err
|
||||||
|
_ffprobe_err="$(echo "$_ffprobe_out" | head -1)"
|
||||||
|
echo "INVALID|ffprobe failed to parse container${_ffprobe_err:+ (${_ffprobe_err})}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
if head -c 64 "$_f" 2> /dev/null | grep -aq "ftyp" ; then
|
if head -c 64 "$_f" 2> /dev/null | grep -aq "ftyp" ; then
|
||||||
echo "VALID|mp4 ftyp box found"
|
echo "VALID|mp4 ftyp box found (ffprobe not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|mp4 ftyp box not found"
|
echo "INVALID|mp4 ftyp box not found"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
doc|xls|ppt|ole|msi)
|
doc|xls|ppt|ole|msi)
|
||||||
local _head8
|
local _head8
|
||||||
@@ -493,6 +633,72 @@ validate_recovered_file() {
|
|||||||
echo "INVALID|GIMP (gimp xcf) signature missing"
|
echo "INVALID|GIMP (gimp xcf) signature missing"
|
||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
|
mp3)
|
||||||
|
if command -v mp3val > /dev/null 2>&1 ; then
|
||||||
|
local _mp3val_out
|
||||||
|
_mp3val_out="$(mp3val "$_f" 2>&1)"
|
||||||
|
if echo "$_mp3val_out" | grep -q 'No errors found' ; then
|
||||||
|
echo "VALID|mp3val: no errors found"
|
||||||
|
else
|
||||||
|
local _mp3_err
|
||||||
|
_mp3_err="$(echo "$_mp3val_out" | grep -iv '^mp3val\|^$' | head -2 | tr '\n' ' ' | sed 's/ $//')"
|
||||||
|
echo "INVALID|mp3val reported errors${_mp3_err:+ (${_mp3_err})}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
local _head3 _head2hex
|
||||||
|
_head3="$(head -c3 "$_f" 2>/dev/null)"
|
||||||
|
_head2hex="$(head -c2 "$_f" 2>/dev/null | od -An -tx1 | tr -d ' \n')"
|
||||||
|
if [[ "$_head3" = "ID3" ]] || echo "$_head2hex" | grep -qE '^fff[bef2]' ; then
|
||||||
|
echo "VALID|MP3 ID3 tag / MPEG sync ok (mp3val not installed — no deep check)"
|
||||||
|
else
|
||||||
|
echo "INVALID|MP3: no ID3 header or MPEG frame sync found"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
flac)
|
||||||
|
if command -v flac > /dev/null 2>&1 ; then
|
||||||
|
local _flac_out _flac_exit
|
||||||
|
_flac_out="$(flac --silent --test "$_f" 2>&1)"
|
||||||
|
_flac_exit=$?
|
||||||
|
if [[ $_flac_exit -eq 0 ]] ; then
|
||||||
|
echo "VALID|flac --test: ok"
|
||||||
|
else
|
||||||
|
local _flac_err
|
||||||
|
_flac_err="$(echo "$_flac_out" | grep -v '^$' | tail -1)"
|
||||||
|
echo "INVALID|flac --test failed${_flac_err:+ (${_flac_err})}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
local _head4hex
|
||||||
|
_head4hex="$(head -c4 "$_f" 2>/dev/null | od -An -tx1 | tr -d ' \n')"
|
||||||
|
if [[ "$_head4hex" = "664c6143" ]] ; then
|
||||||
|
echo "VALID|FLAC 'fLaC' signature ok (flac not installed — no deep check)"
|
||||||
|
else
|
||||||
|
echo "INVALID|FLAC 'fLaC' signature missing"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
ogg|oga|ogv|opus)
|
||||||
|
if command -v ogginfo > /dev/null 2>&1 ; then
|
||||||
|
local _ogg_out _ogg_exit
|
||||||
|
_ogg_out="$(ogginfo "$_f" 2>&1)"
|
||||||
|
_ogg_exit=$?
|
||||||
|
if [[ $_ogg_exit -eq 0 ]] ; then
|
||||||
|
echo "VALID|ogginfo: container parsed ok"
|
||||||
|
else
|
||||||
|
local _ogg_err
|
||||||
|
_ogg_err="$(echo "$_ogg_out" | grep -iv '^Processing\|^$' | head -2 | tr '\n' ' ' | sed 's/ $//')"
|
||||||
|
echo "INVALID|ogginfo failed${_ogg_err:+ (${_ogg_err})}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
local _head4ogg
|
||||||
|
_head4ogg="$(head -c4 "$_f" 2>/dev/null)"
|
||||||
|
if [[ "$_head4ogg" = "OggS" ]] ; then
|
||||||
|
echo "VALID|OGG 'OggS' signature ok (ogginfo not installed — no deep check)"
|
||||||
|
else
|
||||||
|
echo "INVALID|OGG 'OggS' signature missing"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
;;
|
||||||
wav)
|
wav)
|
||||||
local _riff _wave
|
local _riff _wave
|
||||||
_riff="$(head -c4 "$_f" 2> /dev/null)"
|
_riff="$(head -c4 "$_f" 2> /dev/null)"
|
||||||
@@ -564,6 +770,15 @@ validate_recovered_file() {
|
|||||||
esac
|
esac
|
||||||
}
|
}
|
||||||
|
|
||||||
|
## - qpdf exit codes:
|
||||||
|
## - 0 no problems
|
||||||
|
## - 2 structural errors → file is genuinely corrupt/unreadable
|
||||||
|
## - 3 warnings only → minor spec non-conformances; every real
|
||||||
|
## - viewer opens the file without issues
|
||||||
|
## - Only exit code 2 is treated as INVALID here. Exit code 3 (warnings)
|
||||||
|
## - is reported as VALID with a note, preventing false positives for
|
||||||
|
## - real-world PDFs that have trivial non-conformances.
|
||||||
|
## -
|
||||||
_pdf_check() {
|
_pdf_check() {
|
||||||
local _f="$1"
|
local _f="$1"
|
||||||
if [[ "$(head -c5 "$_f" 2> /dev/null)" != "%PDF-" ]] ; then
|
if [[ "$(head -c5 "$_f" 2> /dev/null)" != "%PDF-" ]] ; then
|
||||||
@@ -571,11 +786,22 @@ _pdf_check() {
|
|||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
if command -v qpdf > /dev/null 2>&1 ; then
|
if command -v qpdf > /dev/null 2>&1 ; then
|
||||||
if qpdf --check "$_f" > /dev/null 2>&1 ; then
|
local _qpdf_out _qpdf_exit
|
||||||
|
_qpdf_out="$(qpdf --check "$_f" 2>&1)"
|
||||||
|
_qpdf_exit=$?
|
||||||
|
if [[ $_qpdf_exit -eq 0 ]] ; then
|
||||||
_pdf_check_detail="qpdf --check ok"
|
_pdf_check_detail="qpdf --check ok"
|
||||||
return 0
|
return 0
|
||||||
|
elif [[ $_qpdf_exit -eq 3 ]] ; then
|
||||||
|
# Warnings only — no structural errors. File is readable by all
|
||||||
|
# standard PDF viewers; non-conformances are minor/cosmetic.
|
||||||
|
_pdf_check_detail="qpdf --check ok (warnings only — file is readable)"
|
||||||
|
return 0
|
||||||
else
|
else
|
||||||
_pdf_check_detail="qpdf --check failed"
|
# Exit code 2 (or unexpected): genuine structural errors.
|
||||||
|
local _first_err
|
||||||
|
_first_err="$(echo "$_qpdf_out" | grep -i 'error' | head -1 | sed 's/^[[:space:]]*//')"
|
||||||
|
_pdf_check_detail="qpdf --check failed (exit ${_qpdf_exit}${_first_err:+: ${_first_err}})"
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
@@ -731,7 +957,9 @@ fi
|
|||||||
# -
|
# -
|
||||||
if mkdir "$LOCK_DIR" 2> /dev/null ; then
|
if mkdir "$LOCK_DIR" 2> /dev/null ; then
|
||||||
|
|
||||||
trap clean_up SIGHUP SIGINT SIGTERM
|
# Trap signals AND normal/abnormal exit so that temp files and the
|
||||||
|
# lock directory are always removed — even on an unexpected crash.
|
||||||
|
trap 'clean_up 1' SIGHUP SIGINT SIGTERM EXIT
|
||||||
|
|
||||||
else
|
else
|
||||||
|
|
||||||
@@ -782,8 +1010,11 @@ if ! $_dry_run_explicit && $terminal ; then
|
|||||||
blank_line
|
blank_line
|
||||||
echo -e "\033[37m\033[1mWhich mode should this run use?\033[m"
|
echo -e "\033[37m\033[1mWhich mode should this run use?\033[m"
|
||||||
echo ""
|
echo ""
|
||||||
echo -e " \033[1m[1] Dry-run\033[m - go through everything (selection, re-validation, share check, reporting), but delete, back up, create or verify NOTHING"
|
echo -e " \033[1m[1] Dry-run\033[m - go through everything (selection, re-validation, share check,
|
||||||
echo " [2] Real recreate - actually back up, DELETE the broken file, create it fresh and verify each one"
|
reporting), but delete, back up, create or verify NOTHING"
|
||||||
|
echo ""
|
||||||
|
echo " [2] Real recreate - actually back up, DELETE the broken file, create it fresh
|
||||||
|
and verify each one"
|
||||||
info "Just press Return to use the default: [1] Dry-run."
|
info "Just press Return to use the default: [1] Dry-run."
|
||||||
echo -n " Select mode by number [1]: "
|
echo -n " Select mode by number [1]: "
|
||||||
read _mode_choice
|
read _mode_choice
|
||||||
@@ -1037,6 +1268,8 @@ done
|
|||||||
mkdir -p "$report_dir" 2> /dev/null
|
mkdir -p "$report_dir" 2> /dev/null
|
||||||
recreate_report_file="${report_dir}/recreate_${WEBSITE}_${run_date}.tsv"
|
recreate_report_file="${report_dir}/recreate_${WEBSITE}_${run_date}.tsv"
|
||||||
|
|
||||||
|
check_optional_validation_tools
|
||||||
|
|
||||||
if $terminal ; then
|
if $terminal ; then
|
||||||
echo ""
|
echo ""
|
||||||
if $DRY_RUN ; then
|
if $DRY_RUN ; then
|
||||||
@@ -1080,6 +1313,18 @@ if $terminal ; then
|
|||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
if $terminal ; then
|
||||||
|
echo ""
|
||||||
|
echo -e " \033[1mTemporäre Dateien – werden laufend aktualisiert:\033[m"
|
||||||
|
echo ""
|
||||||
|
echo -e " Fortschritt / Debug-Meldungen (PHP-Ausgabe auf STDERR, pro Account):"
|
||||||
|
echo -e " \033[1mtail -f ${LOCK_DIR}/${script_name%%.*}_<account>.log\033[m"
|
||||||
|
echo ""
|
||||||
|
echo -e " Rohergebnis pro Account (PHP-Ausgabe auf STDOUT, Datei für Datei):"
|
||||||
|
echo -e " \033[1mtail -f ${LOCK_DIR}/result_<account>.tsv\033[m"
|
||||||
|
echo ""
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
||||||
{
|
{
|
||||||
echo "=================================================================="
|
echo "=================================================================="
|
||||||
@@ -1162,6 +1407,13 @@ if ($oldWorkingDir === false) {
|
|||||||
}
|
}
|
||||||
chdir(__DIR__);
|
chdir(__DIR__);
|
||||||
require_once __DIR__ . '/lib/base.php';
|
require_once __DIR__ . '/lib/base.php';
|
||||||
|
// Nextcloud's own bootstrap (lib/base.php) unconditionally calls
|
||||||
|
// set_time_limit(3600) as part of the require above - this OVERRIDES
|
||||||
|
// whatever -d max_execution_time was passed on the PHP command line,
|
||||||
|
// since it is a later, explicit runtime call. Undo that here, now that
|
||||||
|
// the require is done, so recreating a large account isn't killed
|
||||||
|
// after exactly one hour regardless of the CLI flag.
|
||||||
|
set_time_limit(0);
|
||||||
chdir($oldWorkingDir);
|
chdir($oldWorkingDir);
|
||||||
|
|
||||||
if (function_exists('posix_getuid') && posix_getuid() === 0) {
|
if (function_exists('posix_getuid') && posix_getuid() === 0) {
|
||||||
@@ -1763,7 +2015,11 @@ for _user in "${selected_user_arr[@]}" ; do
|
|||||||
_php_args=("$recreate_php_file" "$_user" "$list_file" "$DATA_DIR")
|
_php_args=("$recreate_php_file" "$_user" "$list_file" "$DATA_DIR")
|
||||||
$DRY_RUN && _php_args+=("--dry-run")
|
$DRY_RUN && _php_args+=("--dry-run")
|
||||||
$FORCE_SHARED && _php_args+=("--force")
|
$FORCE_SHARED && _php_args+=("--force")
|
||||||
su -c "$PHP_BIN ${_php_args[*]}" -s /bin/bash $HTTP_USER > "$user_result_tsv" 2> "$log_file"
|
# -d max_execution_time=0: recreating every selected file of a
|
||||||
|
# large account can run well over an hour; without this the PHP
|
||||||
|
# CLI process is killed by PHP's own execution-time limit
|
||||||
|
# (typically inherited from the webserver's php.ini) mid-run.
|
||||||
|
su -c "$PHP_BIN -d max_execution_time=0 ${_php_args[*]}" -s /bin/bash $HTTP_USER > "$user_result_tsv" 2> "$log_file"
|
||||||
_rc=$?
|
_rc=$?
|
||||||
else
|
else
|
||||||
echo -e "path\tbytes_written\tstatus\tdetail" > "$user_result_tsv"
|
echo -e "path\tbytes_written\tstatus\tdetail" > "$user_result_tsv"
|
||||||
|
|||||||
+395
-62
@@ -59,15 +59,14 @@ usage() {
|
|||||||
\033[1mDescription\033[m
|
\033[1mDescription\033[m
|
||||||
|
|
||||||
Writes already-recovered, already-validated files (produced by a
|
Writes already-recovered, already-validated files (produced by a
|
||||||
PRIOR run of recover_bad_signature.sh, validation column = VALID in
|
PRIOR run of recover_bad_signature.sh) back into Nextcloud's own
|
||||||
its report) back into Nextcloud's own storage, at their ORIGINAL
|
storage, at their ORIGINAL path - through Nextcloud's normal Files
|
||||||
path - through Nextcloud's normal Files API (the same write path a
|
API (the same write path a regular upload/sync client write uses),
|
||||||
regular upload/sync client write uses), so the Server-Side
|
so the Server-Side Encryption layer transparently (re-)encrypts the
|
||||||
Encryption layer transparently (re-)encrypts the content with a
|
content with a fresh IV and a correct signature. Afterwards the
|
||||||
fresh IV and a correct signature. Afterwards the file is a
|
file is a completely normal, healthy encrypted file again: readable
|
||||||
completely normal, healthy encrypted file again: readable in the
|
in the web interface, over WebDAV, and by sync clients, with no
|
||||||
web interface, over WebDAV, and by sync clients, with no more 'Bad
|
more 'Bad Signature' and no server-side flag left toggled.
|
||||||
Signature' and no server-side flag left toggled.
|
|
||||||
|
|
||||||
This is the ONLY script in this toolkit that writes into
|
This is the ONLY script in this toolkit that writes into
|
||||||
Nextcloud's live storage - scan_bad_signature.sh and
|
Nextcloud's live storage - scan_bad_signature.sh and
|
||||||
@@ -75,14 +74,35 @@ usage() {
|
|||||||
itself. Treat it accordingly.
|
itself. Treat it accordingly.
|
||||||
|
|
||||||
IMPORTANT:
|
IMPORTANT:
|
||||||
- Only files whose recovery report marks them 'VALID' are ever
|
- Two categories of entries from the recovery report are
|
||||||
considered - 'Datenmuell'/INVALID and 'Nicht pruefbar'/
|
restored, both by default, no separate flag needed:
|
||||||
UNVERIFIED entries are never written back, no override exists.
|
* validation=VALID - a dedicated structural check (PDF
|
||||||
|
header, zip integrity, image signature, ...) actually
|
||||||
|
confirmed the recovered content.
|
||||||
|
* validation=UNVERIFIED ('Nicht pruefbar' in the recovery
|
||||||
|
report) - unusual/unrecognized file type, no dedicated
|
||||||
|
check exists, but nothing about it looks wrong either
|
||||||
|
(recovered/original size ratio is in the expected range,
|
||||||
|
or no size comparison applies to that check path). This
|
||||||
|
is the 'unauffaellig' case: we cannot prove it's correct,
|
||||||
|
but we also have no concrete reason to think it's not.
|
||||||
|
Entries where the size ratio actively 'looks off' (a genuine
|
||||||
|
negative signal, not just 'unchecked') are treated like
|
||||||
|
INVALID and are NEVER restored automatically, same as
|
||||||
|
'Datenmuell'/INVALID entries - no override exists for either.
|
||||||
|
Every restored UNVERIFIED file is clearly marked as such (its
|
||||||
|
own report column, its own counters) so this is always
|
||||||
|
transparent afterwards - never silently indistinguishable from
|
||||||
|
a properly verified VALID restore. If you'd rather review the
|
||||||
|
UNVERIFIED/'Nicht pruefbar' files yourself first, use
|
||||||
|
delete_files.sh to remove the ones you don't want kept, run
|
||||||
|
this script for 'VALID' coverage, then decide on the rest by
|
||||||
|
hand.
|
||||||
- Every file is re-validated with the CURRENT checks right
|
- Every file is re-validated with the CURRENT checks right
|
||||||
before writing (not just trusted from the old report), in case
|
before writing (not just trusted from the old report), in case
|
||||||
this script's validation logic has improved since that report
|
this script's validation logic has improved since that report
|
||||||
was produced - files that no longer validate are skipped, not
|
was produced - files that no longer validate (as VALID or as
|
||||||
forced through.
|
plausible UNVERIFIED) are skipped, not forced through.
|
||||||
- Before each file is overwritten, the CURRENT on-disk ciphertext
|
- Before each file is overwritten, the CURRENT on-disk ciphertext
|
||||||
(the still-broken original, exactly as it is right now) is
|
(the still-broken original, exactly as it is right now) is
|
||||||
copied byte-for-byte to a separate backup directory (default
|
copied byte-for-byte to a separate backup directory (default
|
||||||
@@ -113,7 +133,7 @@ usage() {
|
|||||||
reports what would happen) or the real thing. You will then be
|
reports what would happen) or the real thing. You will then be
|
||||||
asked which existing recovery report (produced by
|
asked which existing recovery report (produced by
|
||||||
recover_bad_signature.sh) to use, and then which account(s) from its
|
recover_bad_signature.sh) to use, and then which account(s) from its
|
||||||
VALID entries to restore for:
|
restorable (VALID + plausible UNVERIFIED) entries to restore for:
|
||||||
|
|
||||||
- a blank separated list of account names
|
- a blank separated list of account names
|
||||||
- or 'all' to attempt every account found in the report
|
- or 'all' to attempt every account found in the report
|
||||||
@@ -147,6 +167,8 @@ usage() {
|
|||||||
clean_up() {
|
clean_up() {
|
||||||
|
|
||||||
# Perform program exit housekeeping
|
# Perform program exit housekeeping
|
||||||
|
# Clear EXIT trap first so that the exit below does not fire it again.
|
||||||
|
trap - EXIT
|
||||||
[[ -n "$restore_php_file" ]] && rm -f "$restore_php_file" 2> /dev/null
|
[[ -n "$restore_php_file" ]] && rm -f "$restore_php_file" 2> /dev/null
|
||||||
rm -rf "$LOCK_DIR"
|
rm -rf "$LOCK_DIR"
|
||||||
blank_line
|
blank_line
|
||||||
@@ -154,6 +176,77 @@ clean_up() {
|
|||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
check_optional_validation_tools() {
|
||||||
|
|
||||||
|
local _missing_tools=()
|
||||||
|
local _missing_pkgs=()
|
||||||
|
local _desc=()
|
||||||
|
|
||||||
|
command -v identify > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("identify")
|
||||||
|
_missing_pkgs+=("imagemagick")
|
||||||
|
_desc+=("identify (imagemagick) — PNG / TIFF / BMP / GIF: vollst. Dekodierung statt nur Magic-Bytes")
|
||||||
|
}
|
||||||
|
command -v ffprobe > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("ffprobe")
|
||||||
|
_missing_pkgs+=("ffmpeg")
|
||||||
|
_desc+=("ffprobe (ffmpeg) — MP4 / MOV / M4V: Container-Parsing statt nur ftyp-Box-Suche")
|
||||||
|
}
|
||||||
|
command -v mp3val > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("mp3val")
|
||||||
|
_missing_pkgs+=("mp3val")
|
||||||
|
_desc+=("mp3val (mp3val) — MP3: Frame-Struktur-Check statt nur ID3-Signatur")
|
||||||
|
}
|
||||||
|
command -v flac > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("flac")
|
||||||
|
_missing_pkgs+=("flac")
|
||||||
|
_desc+=("flac (flac) — FLAC: Decode-Test statt nur fLaC-Signatur")
|
||||||
|
}
|
||||||
|
command -v ogginfo > /dev/null 2>&1 || {
|
||||||
|
_missing_tools+=("ogginfo")
|
||||||
|
_missing_pkgs+=("vorbis-tools")
|
||||||
|
_desc+=("ogginfo (vorbis-tools) — OGG / OGA / OPUS: Container-Check statt nur OggS-Signatur")
|
||||||
|
}
|
||||||
|
|
||||||
|
[[ ${#_missing_tools[@]} -eq 0 ]] && return 0
|
||||||
|
|
||||||
|
if ! $terminal ; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
local _pkg_list="${_missing_pkgs[*]}"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e " \033[33mOptionale Validierungs-Tools fehlen – Checks laufen mit reduzierter Genauigkeit:\033[m"
|
||||||
|
echo ""
|
||||||
|
local _d
|
||||||
|
for _d in "${_desc[@]}" ; do
|
||||||
|
echo " ${_d}"
|
||||||
|
done
|
||||||
|
echo ""
|
||||||
|
echo -n " Jetzt installieren? apt install ${_pkg_list} [j/N]: "
|
||||||
|
read -r _yn
|
||||||
|
echo ""
|
||||||
|
if [[ "$_yn" =~ ^[jJyY]$ ]] ; then
|
||||||
|
echononl " Installiere Pakete: ${_pkg_list}.."
|
||||||
|
# shellcheck disable=SC2086
|
||||||
|
if apt-get install -y ${_missing_pkgs[*]} > /dev/null 2>&1 ; then
|
||||||
|
echo_ok
|
||||||
|
else
|
||||||
|
echo_failed
|
||||||
|
echo ""
|
||||||
|
echo " Bitte manuell installieren:"
|
||||||
|
echo " apt install ${_pkg_list}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo " Übersprungen — Validierung läuft mit reduzierter Genauigkeit."
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
is_number() {
|
is_number() {
|
||||||
|
|
||||||
return $(test ! -z "${1##*[!0-9]*}" > /dev/null 2>&1);
|
return $(test ! -z "${1##*[!0-9]*}" > /dev/null 2>&1);
|
||||||
@@ -303,38 +396,70 @@ validate_recovered_file() {
|
|||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
png)
|
png)
|
||||||
|
if command -v identify > /dev/null 2>&1 ; then
|
||||||
|
if identify -regard-warnings "$_f" > /dev/null 2>&1 ; then
|
||||||
|
echo "VALID|ImageMagick 'identify' decoded PNG ok"
|
||||||
|
else
|
||||||
|
echo "INVALID|ImageMagick 'identify' failed to decode PNG"
|
||||||
|
fi
|
||||||
|
else
|
||||||
local _head
|
local _head
|
||||||
_head="$(head -c8 "$_f" 2> /dev/null | od -An -tx1 | tr -d ' \n')"
|
_head="$(head -c8 "$_f" 2> /dev/null | od -An -tx1 | tr -d ' \n')"
|
||||||
if [[ "$_head" = "89504e470d0a1a0a" ]] ; then
|
if [[ "$_head" = "89504e470d0a1a0a" ]] ; then
|
||||||
echo "VALID|PNG signature ok"
|
echo "VALID|PNG signature ok (identify not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|PNG signature missing"
|
echo "INVALID|PNG signature missing"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
gif)
|
gif)
|
||||||
|
if command -v identify > /dev/null 2>&1 ; then
|
||||||
|
if identify -regard-warnings "$_f" > /dev/null 2>&1 ; then
|
||||||
|
echo "VALID|ImageMagick 'identify' decoded GIF ok"
|
||||||
|
else
|
||||||
|
echo "INVALID|ImageMagick 'identify' failed to decode GIF"
|
||||||
|
fi
|
||||||
|
else
|
||||||
local _head6
|
local _head6
|
||||||
_head6="$(head -c6 "$_f" 2> /dev/null)"
|
_head6="$(head -c6 "$_f" 2> /dev/null)"
|
||||||
if [[ "$_head6" = "GIF87a" || "$_head6" = "GIF89a" ]] ; then
|
if [[ "$_head6" = "GIF87a" || "$_head6" = "GIF89a" ]] ; then
|
||||||
echo "VALID|GIF signature ok"
|
echo "VALID|GIF signature ok (identify not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|GIF signature missing"
|
echo "INVALID|GIF signature missing"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
bmp)
|
bmp)
|
||||||
|
if command -v identify > /dev/null 2>&1 ; then
|
||||||
|
if identify -regard-warnings "$_f" > /dev/null 2>&1 ; then
|
||||||
|
echo "VALID|ImageMagick 'identify' decoded BMP ok"
|
||||||
|
else
|
||||||
|
echo "INVALID|ImageMagick 'identify' failed to decode BMP"
|
||||||
|
fi
|
||||||
|
else
|
||||||
if [[ "$(head -c2 "$_f" 2> /dev/null)" = "BM" ]] ; then
|
if [[ "$(head -c2 "$_f" 2> /dev/null)" = "BM" ]] ; then
|
||||||
echo "VALID|BMP signature ok"
|
echo "VALID|BMP signature ok (identify not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|BMP signature missing"
|
echo "INVALID|BMP signature missing"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
tif|tiff)
|
tif|tiff)
|
||||||
|
if command -v identify > /dev/null 2>&1 ; then
|
||||||
|
if identify -regard-warnings "$_f" > /dev/null 2>&1 ; then
|
||||||
|
echo "VALID|ImageMagick 'identify' decoded TIFF ok"
|
||||||
|
else
|
||||||
|
echo "INVALID|ImageMagick 'identify' failed to decode TIFF"
|
||||||
|
fi
|
||||||
|
else
|
||||||
local _head4
|
local _head4
|
||||||
_head4="$(head -c4 "$_f" 2> /dev/null | od -An -tx1 | tr -d ' \n')"
|
_head4="$(head -c4 "$_f" 2> /dev/null | od -An -tx1 | tr -d ' \n')"
|
||||||
if [[ "$_head4" = "49492a00" || "$_head4" = "4d4d002a" ]] ; then
|
if [[ "$_head4" = "49492a00" || "$_head4" = "4d4d002a" ]] ; then
|
||||||
echo "VALID|TIFF signature ok"
|
echo "VALID|TIFF signature ok (identify not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|TIFF signature missing"
|
echo "INVALID|TIFF signature missing"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
pnm|pgm|ppm|pbm)
|
pnm|pgm|ppm|pbm)
|
||||||
local _head2
|
local _head2
|
||||||
@@ -360,23 +485,58 @@ validate_recovered_file() {
|
|||||||
if [[ "$_head8o" = "d0cf11e0a1b11ae1" ]] ; then
|
if [[ "$_head8o" = "d0cf11e0a1b11ae1" ]] ; then
|
||||||
echo "VALID|OLE2/CFBF container signature ok - this is a password-protected Office file (encrypted package), not a plain zip, so the zip check does not apply; open it with the password to verify content"
|
echo "VALID|OLE2/CFBF container signature ok - this is a password-protected Office file (encrypted package), not a plain zip, so the zip check does not apply; open it with the password to verify content"
|
||||||
elif command -v unzip > /dev/null 2>&1 ; then
|
elif command -v unzip > /dev/null 2>&1 ; then
|
||||||
if unzip -tq "$_f" > /dev/null 2>&1 ; then
|
# Run unzip inside setsid so it has no controlling terminal.
|
||||||
|
# Without setsid, a password-protected ZIP causes unzip to try
|
||||||
|
# opening /dev/tty to prompt for the password. When running in
|
||||||
|
# the background of a tmux session this generates SIGTTIN, which
|
||||||
|
# STOPS the process (ps state T). A stopped process cannot receive
|
||||||
|
# SIGTERM, so timeout hangs indefinitely waiting for a child that
|
||||||
|
# will never exit. With setsid the /dev/tty open fails immediately
|
||||||
|
# with ENXIO and unzip exits with a non-zero code instead.
|
||||||
|
# -k 5: send SIGKILL 5 s after SIGTERM in case the process is
|
||||||
|
# still alive (e.g. stopped or in uninterruptible sleep).
|
||||||
|
local _unzip_exit
|
||||||
|
timeout -k 5 120 setsid unzip -tq "$_f" < /dev/null > /dev/null 2>&1
|
||||||
|
_unzip_exit=$?
|
||||||
|
if [[ $_unzip_exit -eq 0 ]] ; then
|
||||||
echo "VALID|zip integrity ok"
|
echo "VALID|zip integrity ok"
|
||||||
|
elif [[ $_unzip_exit -eq 124 || $_unzip_exit -eq 137 ]] ; then
|
||||||
|
# 124 = killed by SIGTERM after timeout, 137 = killed by SIGKILL (128+9)
|
||||||
|
echo "UNVERIFIED|zip integrity check timed out after 120 s (file may be very large or corrupt; check manually with: unzip -t \"$_f\")"
|
||||||
else
|
else
|
||||||
local _badentry
|
local _badentry
|
||||||
_badentry="$(trim "$(unzip -t "$_f" 2>&1 | grep -v '^Archive:' | grep -v '^[[:space:]]*$' | head -1)")"
|
_badentry="$(trim "$(timeout -k 5 120 setsid unzip -t "$_f" < /dev/null 2>&1 | grep -v '^Archive:' | grep -v '^[[:space:]]*$' | head -1)")"
|
||||||
|
if echo "$_badentry" | grep -qi "password\|encrypt\|need PK compat" ; then
|
||||||
|
echo "UNVERIFIED|zip is password-protected (cannot verify without password${_badentry:+; unzip says: ${_badentry}})"
|
||||||
|
else
|
||||||
echo "INVALID|zip integrity check failed${_badentry:+ (${_badentry})}"
|
echo "INVALID|zip integrity check failed${_badentry:+ (${_badentry})}"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
else
|
else
|
||||||
echo "UNVERIFIED|unzip not installed"
|
echo "UNVERIFIED|unzip not installed"
|
||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
mp4|mov|m4v)
|
mp4|mov|m4v)
|
||||||
|
if command -v ffprobe > /dev/null 2>&1 ; then
|
||||||
|
local _ffprobe_out _ffprobe_exit
|
||||||
|
_ffprobe_out="$(ffprobe -v error -show_streams "$_f" 2>&1)"
|
||||||
|
_ffprobe_exit=$?
|
||||||
|
if [[ $_ffprobe_exit -eq 0 ]] ; then
|
||||||
|
local _streams
|
||||||
|
_streams="$(echo "$_ffprobe_out" | grep -c '\[STREAM\]' || true)"
|
||||||
|
echo "VALID|ffprobe parsed container ok (${_streams} stream(s) found)"
|
||||||
|
else
|
||||||
|
local _ffprobe_err
|
||||||
|
_ffprobe_err="$(echo "$_ffprobe_out" | head -1)"
|
||||||
|
echo "INVALID|ffprobe failed to parse container${_ffprobe_err:+ (${_ffprobe_err})}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
if head -c 64 "$_f" 2> /dev/null | grep -aq "ftyp" ; then
|
if head -c 64 "$_f" 2> /dev/null | grep -aq "ftyp" ; then
|
||||||
echo "VALID|mp4 ftyp box found"
|
echo "VALID|mp4 ftyp box found (ffprobe not installed — no deep check)"
|
||||||
else
|
else
|
||||||
echo "INVALID|mp4 ftyp box not found"
|
echo "INVALID|mp4 ftyp box not found"
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
doc|xls|ppt|ole|msi)
|
doc|xls|ppt|ole|msi)
|
||||||
local _head8
|
local _head8
|
||||||
@@ -475,6 +635,72 @@ validate_recovered_file() {
|
|||||||
echo "INVALID|GIMP (gimp xcf) signature missing"
|
echo "INVALID|GIMP (gimp xcf) signature missing"
|
||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
|
mp3)
|
||||||
|
if command -v mp3val > /dev/null 2>&1 ; then
|
||||||
|
local _mp3val_out
|
||||||
|
_mp3val_out="$(mp3val "$_f" 2>&1)"
|
||||||
|
if echo "$_mp3val_out" | grep -q 'No errors found' ; then
|
||||||
|
echo "VALID|mp3val: no errors found"
|
||||||
|
else
|
||||||
|
local _mp3_err
|
||||||
|
_mp3_err="$(echo "$_mp3val_out" | grep -iv '^mp3val\|^$' | head -2 | tr '\n' ' ' | sed 's/ $//')"
|
||||||
|
echo "INVALID|mp3val reported errors${_mp3_err:+ (${_mp3_err})}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
local _head3 _head2hex
|
||||||
|
_head3="$(head -c3 "$_f" 2>/dev/null)"
|
||||||
|
_head2hex="$(head -c2 "$_f" 2>/dev/null | od -An -tx1 | tr -d ' \n')"
|
||||||
|
if [[ "$_head3" = "ID3" ]] || echo "$_head2hex" | grep -qE '^fff[bef2]' ; then
|
||||||
|
echo "VALID|MP3 ID3 tag / MPEG sync ok (mp3val not installed — no deep check)"
|
||||||
|
else
|
||||||
|
echo "INVALID|MP3: no ID3 header or MPEG frame sync found"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
flac)
|
||||||
|
if command -v flac > /dev/null 2>&1 ; then
|
||||||
|
local _flac_out _flac_exit
|
||||||
|
_flac_out="$(flac --silent --test "$_f" 2>&1)"
|
||||||
|
_flac_exit=$?
|
||||||
|
if [[ $_flac_exit -eq 0 ]] ; then
|
||||||
|
echo "VALID|flac --test: ok"
|
||||||
|
else
|
||||||
|
local _flac_err
|
||||||
|
_flac_err="$(echo "$_flac_out" | grep -v '^$' | tail -1)"
|
||||||
|
echo "INVALID|flac --test failed${_flac_err:+ (${_flac_err})}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
local _head4hex
|
||||||
|
_head4hex="$(head -c4 "$_f" 2>/dev/null | od -An -tx1 | tr -d ' \n')"
|
||||||
|
if [[ "$_head4hex" = "664c6143" ]] ; then
|
||||||
|
echo "VALID|FLAC 'fLaC' signature ok (flac not installed — no deep check)"
|
||||||
|
else
|
||||||
|
echo "INVALID|FLAC 'fLaC' signature missing"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
ogg|oga|ogv|opus)
|
||||||
|
if command -v ogginfo > /dev/null 2>&1 ; then
|
||||||
|
local _ogg_out _ogg_exit
|
||||||
|
_ogg_out="$(ogginfo "$_f" 2>&1)"
|
||||||
|
_ogg_exit=$?
|
||||||
|
if [[ $_ogg_exit -eq 0 ]] ; then
|
||||||
|
echo "VALID|ogginfo: container parsed ok"
|
||||||
|
else
|
||||||
|
local _ogg_err
|
||||||
|
_ogg_err="$(echo "$_ogg_out" | grep -iv '^Processing\|^$' | head -2 | tr '\n' ' ' | sed 's/ $//')"
|
||||||
|
echo "INVALID|ogginfo failed${_ogg_err:+ (${_ogg_err})}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
local _head4ogg
|
||||||
|
_head4ogg="$(head -c4 "$_f" 2>/dev/null)"
|
||||||
|
if [[ "$_head4ogg" = "OggS" ]] ; then
|
||||||
|
echo "VALID|OGG 'OggS' signature ok (ogginfo not installed — no deep check)"
|
||||||
|
else
|
||||||
|
echo "INVALID|OGG 'OggS' signature missing"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
;;
|
||||||
wav)
|
wav)
|
||||||
local _riff _wave
|
local _riff _wave
|
||||||
_riff="$(head -c4 "$_f" 2> /dev/null)"
|
_riff="$(head -c4 "$_f" 2> /dev/null)"
|
||||||
@@ -562,6 +788,15 @@ validate_recovered_file() {
|
|||||||
## - PDF structural check shared by the 'pdf' case and the 'ai'
|
## - PDF structural check shared by the 'pdf' case and the 'ai'
|
||||||
## - (PDF-compatible) case. Sets $_pdf_check_detail, returns 0/1.
|
## - (PDF-compatible) case. Sets $_pdf_check_detail, returns 0/1.
|
||||||
## -
|
## -
|
||||||
|
## - qpdf exit codes:
|
||||||
|
## - 0 no problems
|
||||||
|
## - 2 structural errors → file is genuinely corrupt/unreadable
|
||||||
|
## - 3 warnings only → minor spec non-conformances; every real
|
||||||
|
## - viewer opens the file without issues
|
||||||
|
## - Only exit code 2 is treated as INVALID here. Exit code 3 (warnings)
|
||||||
|
## - is reported as VALID with a note, preventing false positives for
|
||||||
|
## - real-world PDFs that have trivial non-conformances.
|
||||||
|
## -
|
||||||
_pdf_check() {
|
_pdf_check() {
|
||||||
local _f="$1"
|
local _f="$1"
|
||||||
if [[ "$(head -c5 "$_f" 2> /dev/null)" != "%PDF-" ]] ; then
|
if [[ "$(head -c5 "$_f" 2> /dev/null)" != "%PDF-" ]] ; then
|
||||||
@@ -569,11 +804,22 @@ _pdf_check() {
|
|||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
if command -v qpdf > /dev/null 2>&1 ; then
|
if command -v qpdf > /dev/null 2>&1 ; then
|
||||||
if qpdf --check "$_f" > /dev/null 2>&1 ; then
|
local _qpdf_out _qpdf_exit
|
||||||
|
_qpdf_out="$(qpdf --check "$_f" 2>&1)"
|
||||||
|
_qpdf_exit=$?
|
||||||
|
if [[ $_qpdf_exit -eq 0 ]] ; then
|
||||||
_pdf_check_detail="qpdf --check ok"
|
_pdf_check_detail="qpdf --check ok"
|
||||||
return 0
|
return 0
|
||||||
|
elif [[ $_qpdf_exit -eq 3 ]] ; then
|
||||||
|
# Warnings only — no structural errors. File is readable by all
|
||||||
|
# standard PDF viewers; non-conformances are minor/cosmetic.
|
||||||
|
_pdf_check_detail="qpdf --check ok (warnings only — file is readable)"
|
||||||
|
return 0
|
||||||
else
|
else
|
||||||
_pdf_check_detail="qpdf --check failed"
|
# Exit code 2 (or unexpected): genuine structural errors.
|
||||||
|
local _first_err
|
||||||
|
_first_err="$(echo "$_qpdf_out" | grep -i 'error' | head -1 | sed 's/^[[:space:]]*//')"
|
||||||
|
_pdf_check_detail="qpdf --check failed (exit ${_qpdf_exit}${_first_err:+: ${_first_err}})"
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
@@ -741,7 +987,9 @@ if mkdir "$LOCK_DIR" 2> /dev/null ; then
|
|||||||
|
|
||||||
# - Remove lockdir when the script finishes, or when it receives a signal
|
# - Remove lockdir when the script finishes, or when it receives a signal
|
||||||
# -
|
# -
|
||||||
trap clean_up SIGHUP SIGINT SIGTERM
|
# Trap signals AND normal/abnormal exit so that temp files and the
|
||||||
|
# lock directory are always removed — even on an unexpected crash.
|
||||||
|
trap 'clean_up 1' SIGHUP SIGINT SIGTERM EXIT
|
||||||
|
|
||||||
else
|
else
|
||||||
|
|
||||||
@@ -792,7 +1040,9 @@ if ! $_dry_run_explicit && $terminal ; then
|
|||||||
blank_line
|
blank_line
|
||||||
echo -e "\033[37m\033[1mWhich mode should this run use?\033[m"
|
echo -e "\033[37m\033[1mWhich mode should this run use?\033[m"
|
||||||
echo ""
|
echo ""
|
||||||
echo -e " \033[1m[1] Dry-run\033[m - go through everything (selection, re-validation, reporting), but write, back up and verify NOTHING"
|
echo -e " \033[1m[1] Dry-run\033[m - go through everything (selection, re-validation, reporting),
|
||||||
|
but write, back up and verify NOTHING"
|
||||||
|
echo ""
|
||||||
echo " [2] Real restore - actually back up, overwrite and verify each file in Nextcloud"
|
echo " [2] Real restore - actually back up, overwrite and verify each file in Nextcloud"
|
||||||
info "Just press Return to use the default: [1] Dry-run."
|
info "Just press Return to use the default: [1] Dry-run."
|
||||||
echo -n " Select mode by number [1]: "
|
echo -n " Select mode by number [1]: "
|
||||||
@@ -900,6 +1150,7 @@ if $terminal ; then
|
|||||||
echo -e "Restore already-recovered \033[1mBad Signature\033[m files back into system \033[1m${WEB_BASE_DIR}\033[m"
|
echo -e "Restore already-recovered \033[1mBad Signature\033[m files back into system \033[1m${WEB_BASE_DIR}\033[m"
|
||||||
echo -e "\033[1m
|
echo -e "\033[1m
|
||||||
This WRITES into Nextcloud's live storage. Only files marked VALID
|
This WRITES into Nextcloud's live storage. Only files marked VALID
|
||||||
|
or plausible/UNVERIFIED ('Nicht pruefbar', but nothing looks wrong)
|
||||||
by a prior recover_bad_signature.sh run are ever restored, each is
|
by a prior recover_bad_signature.sh run are ever restored, each is
|
||||||
re-validated again right before writing, the current (still broken)
|
re-validated again right before writing, the current (still broken)
|
||||||
original is backed up first, and every write is verified by reading
|
original is backed up first, and every write is verified by reading
|
||||||
@@ -999,12 +1250,26 @@ chosen_report="${_available_reports[$((_report_choice-1))]}"
|
|||||||
|
|
||||||
|
|
||||||
# =============
|
# =============
|
||||||
# --- Extract "user<TAB>path" pairs for every VALID entry in the chosen
|
# --- Extract "user<TAB>path<TAB>original_size<TAB>validation" quadruples
|
||||||
# --- report. The report has no per-row user column - accounts are
|
# --- for every restorable entry in the chosen report. The report has no
|
||||||
# --- section headers ('---------' / '<user>' / '---------'). Only
|
# --- per-row user column - accounts are section headers ('---------' /
|
||||||
# --- 6-tab-field data rows belong to the following case; the header
|
# --- '<user>' / '---------'). Only 6-tab-field data rows belong to the
|
||||||
# --- row, the 'Datenmuell'/'Nicht pruefbar' detail blocks (2 fields)
|
# --- following case; the header row, the 'Datenmuell'/'Nicht pruefbar'
|
||||||
# --- and the account-statistics lines (no tabs) never match NF==6.
|
# --- detail blocks (2 fields) and the account-statistics lines (no
|
||||||
|
# --- tabs) never match NF==6.
|
||||||
|
# ---
|
||||||
|
# --- Two validation values are picked up, both restored by default:
|
||||||
|
# --- VALID - a dedicated structural check confirmed the file.
|
||||||
|
# --- UNVERIFIED - no dedicated check exists ('Nicht pruefbar' in the
|
||||||
|
# --- recovery report), but nothing about it looks
|
||||||
|
# --- actively wrong either. The one sub-case that IS a
|
||||||
|
# --- concrete negative signal - the detail text says
|
||||||
|
# --- the recovered/original size ratio 'LOOKS OFF' -
|
||||||
|
# --- is deliberately excluded here, same treatment as
|
||||||
|
# --- INVALID/'Datenmuell': not restored automatically.
|
||||||
|
# --- original_size ($3) is carried through so this same distinction
|
||||||
|
# --- (plausible size ratio or not) can be re-checked with real data
|
||||||
|
# --- right before writing, not just trusted from the old report.
|
||||||
# =============
|
# =============
|
||||||
|
|
||||||
valid_entries_file="${LOCK_DIR}/valid_entries.tsv"
|
valid_entries_file="${LOCK_DIR}/valid_entries.tsv"
|
||||||
@@ -1016,8 +1281,8 @@ awk -F'\t' '
|
|||||||
}
|
}
|
||||||
{
|
{
|
||||||
if (state == 1) { user = $0; state = 2; next }
|
if (state == 1) { user = $0; state = 2; next }
|
||||||
if (NF == 6 && $1 != "path" && $5 == "VALID") {
|
if (NF == 6 && $1 != "path" && ($5 == "VALID" || ($5 == "UNVERIFIED" && $6 !~ /LOOKS OFF/))) {
|
||||||
print user "\t" $1
|
print user "\t" $1 "\t" $3 "\t" $5
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
' "$chosen_report" > "$valid_entries_file"
|
' "$chosen_report" > "$valid_entries_file"
|
||||||
@@ -1027,21 +1292,26 @@ IFS=$'\n' account_arr=($(sort <<<"${unsorted_account_arr[*]}"))
|
|||||||
IFS=$CUR_IFS
|
IFS=$CUR_IFS
|
||||||
|
|
||||||
if [[ ${#account_arr[@]} -eq 0 ]] ; then
|
if [[ ${#account_arr[@]} -eq 0 ]] ; then
|
||||||
fatal "No 'VALID' entries found in '$(basename "$chosen_report")' - nothing to restore."
|
fatal "No restorable (VALID or plausible UNVERIFIED) entries found in '$(basename "$chosen_report")' - nothing to restore."
|
||||||
fi
|
fi
|
||||||
|
|
||||||
blank_line
|
blank_line
|
||||||
if $terminal ; then
|
if $terminal ; then
|
||||||
echo -e "\033[37m\033[1mAccounts with 'VALID' entries in this report\033[m"
|
echo -e "\033[37m\033[1mAccounts with restorable (VALID + plausible UNVERIFIED) entries in this report\033[m"
|
||||||
echo ""
|
echo ""
|
||||||
for _a in "${account_arr[@]}" ; do
|
for _a in "${account_arr[@]}" ; do
|
||||||
_a_count=$(awk -F'\t' -v u="$_a" '$1==u' "$valid_entries_file" | wc -l)
|
_a_count=$(awk -F'\t' -v u="$_a" '$1==u' "$valid_entries_file" | wc -l)
|
||||||
|
_a_count_unverified=$(awk -F'\t' -v u="$_a" '$1==u && $4=="UNVERIFIED"' "$valid_entries_file" | wc -l)
|
||||||
|
if [[ "$_a_count_unverified" -gt 0 ]] ; then
|
||||||
|
echo " - $_a (${_a_count}, davon ${_a_count_unverified} 'Nicht pruefbar'/plausibel)"
|
||||||
|
else
|
||||||
echo " - $_a (${_a_count})"
|
echo " - $_a (${_a_count})"
|
||||||
|
fi
|
||||||
done
|
done
|
||||||
echo ""
|
echo ""
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo -n " Account(s) to restore VALID files for (blank separated, or 'all'): "
|
echo -n " Account(s) to restore VALID/plausible-UNVERIFIED files for (blank separated, or 'all'): "
|
||||||
read _input
|
read _input
|
||||||
|
|
||||||
while true ; do
|
while true ; do
|
||||||
@@ -1067,7 +1337,7 @@ while true ; do
|
|||||||
break
|
break
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo -n " Account(s) to restore VALID files for (blank separated, or 'all'): "
|
echo -n " Account(s) to restore VALID/plausible-UNVERIFIED files for (blank separated, or 'all'): "
|
||||||
read _input
|
read _input
|
||||||
|
|
||||||
done
|
done
|
||||||
@@ -1080,6 +1350,8 @@ done
|
|||||||
mkdir -p "$report_dir" 2> /dev/null
|
mkdir -p "$report_dir" 2> /dev/null
|
||||||
restore_report_file="${report_dir}/restore_${WEBSITE}_${run_date}.tsv"
|
restore_report_file="${report_dir}/restore_${WEBSITE}_${run_date}.tsv"
|
||||||
|
|
||||||
|
check_optional_validation_tools
|
||||||
|
|
||||||
if $terminal ; then
|
if $terminal ; then
|
||||||
echo ""
|
echo ""
|
||||||
if $DRY_RUN ; then
|
if $DRY_RUN ; then
|
||||||
@@ -1106,7 +1378,7 @@ if $terminal ; then
|
|||||||
if $DRY_RUN ; then
|
if $DRY_RUN ; then
|
||||||
info "Dry-run: selection and re-validation run for real, nothing is backed up, written or verified."
|
info "Dry-run: selection and re-validation run for real, nothing is backed up, written or verified."
|
||||||
else
|
else
|
||||||
warn "This WRITES into Nextcloud's live storage, overwriting the still-broken original at its ORIGINAL path with the recovered content - the first script in this toolkit that does so. Only files re-validated as VALID right now are touched. The current (still broken) original is copied byte-for-byte to '${backup_dir}' first, and every write is read back and verified afterwards, but please still spot-check a few results yourself."
|
warn "This WRITES into Nextcloud's live storage, overwriting the still-broken original at its ORIGINAL path with the recovered content - the first script in this toolkit that does so. Only files re-validated as VALID, or as plausible UNVERIFIED (no dedicated check, but nothing looks wrong - clearly marked as such in the report), right now are touched. The current (still broken) original is copied byte-for-byte to '${backup_dir}' first, and every write is read back and verified afterwards, but please still spot-check a few results yourself - especially the UNVERIFIED ones."
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
@@ -1115,12 +1387,24 @@ if $terminal ; then
|
|||||||
if [[ "$OK" = "YES" ]] ; then
|
if [[ "$OK" = "YES" ]] ; then
|
||||||
echo ""
|
echo ""
|
||||||
echo ""
|
echo ""
|
||||||
echo -e "\033[1;32mGoing to restore VALID files for each selected account on \033[1;37m$WEBSITE \033[m"
|
echo -e "\033[1;32mGoing to restore VALID/plausible-UNVERIFIED files for each selected account on \033[1;37m$WEBSITE \033[m"
|
||||||
else
|
else
|
||||||
fatal "Abort by user request - Answer as not 'YES'"
|
fatal "Abort by user request - Answer as not 'YES'"
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
if $terminal ; then
|
||||||
|
echo ""
|
||||||
|
echo -e " \033[1mTemporäre Dateien – werden laufend aktualisiert:\033[m"
|
||||||
|
echo ""
|
||||||
|
echo -e " Fortschritt / Debug-Meldungen (PHP-Ausgabe auf STDERR):"
|
||||||
|
echo -e " \033[1mtail -f ${log_file}\033[m"
|
||||||
|
echo ""
|
||||||
|
echo -e " Rohergebnis pro Account (PHP-Ausgabe auf STDOUT, Datei für Datei):"
|
||||||
|
echo -e " \033[1mtail -f ${LOCK_DIR}/result_<account>.tsv\033[m"
|
||||||
|
echo ""
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
||||||
{
|
{
|
||||||
echo "=================================================================="
|
echo "=================================================================="
|
||||||
@@ -1128,11 +1412,13 @@ fi
|
|||||||
echo "=================================================================="
|
echo "=================================================================="
|
||||||
echo ""
|
echo ""
|
||||||
echo " Basis-Report (Recovery): $(basename "$chosen_report")"
|
echo " Basis-Report (Recovery): $(basename "$chosen_report")"
|
||||||
echo " Nur Eintraege mit validation=VALID aus diesem Report werden zurueckgeschrieben."
|
echo " Eintraege mit validation=VALID sowie validation=UNVERIFIED ('Nicht pruefbar',"
|
||||||
|
echo " aber Groessenverhaeltnis plausibel) aus diesem Report werden zurueckgeschrieben -"
|
||||||
|
echo " letztere sind in der Spalte 'origin' unten als UNVERIFIED gekennzeichnet."
|
||||||
$DRY_RUN && echo " DRY-RUN: es wurde NICHTS geschrieben, gesichert oder verifiziert."
|
$DRY_RUN && echo " DRY-RUN: es wurde NICHTS geschrieben, gesichert oder verifiziert."
|
||||||
! $DRY_RUN && echo " Chiffretext-Sicherung (vor dem Ueberschreiben) liegt unter: ${backup_dir}"
|
! $DRY_RUN && echo " Chiffretext-Sicherung (vor dem Ueberschreiben) liegt unter: ${backup_dir}"
|
||||||
echo ""
|
echo ""
|
||||||
echo -e "path\tbytes_written\tstatus\tdetail"
|
echo -e "path\tbytes_written\tstatus\torigin\tdetail"
|
||||||
} > "$restore_report_file"
|
} > "$restore_report_file"
|
||||||
|
|
||||||
|
|
||||||
@@ -1183,6 +1469,13 @@ if ($oldWorkingDir === false) {
|
|||||||
}
|
}
|
||||||
chdir(__DIR__);
|
chdir(__DIR__);
|
||||||
require_once __DIR__ . '/lib/base.php';
|
require_once __DIR__ . '/lib/base.php';
|
||||||
|
// Nextcloud's own bootstrap (lib/base.php) unconditionally calls
|
||||||
|
// set_time_limit(3600) as part of the require above - this OVERRIDES
|
||||||
|
// whatever -d max_execution_time was passed on the PHP command line,
|
||||||
|
// since it is a later, explicit runtime call. Undo that here, now that
|
||||||
|
// the require is done, so restoring a large account isn't killed
|
||||||
|
// after exactly one hour regardless of the CLI flag.
|
||||||
|
set_time_limit(0);
|
||||||
chdir($oldWorkingDir);
|
chdir($oldWorkingDir);
|
||||||
|
|
||||||
if (function_exists('posix_getuid') && posix_getuid() === 0) {
|
if (function_exists('posix_getuid') && posix_getuid() === 0) {
|
||||||
@@ -1356,8 +1649,10 @@ if ! $DRY_RUN ; then
|
|||||||
fi
|
fi
|
||||||
|
|
||||||
declare -i total_selected=0
|
declare -i total_selected=0
|
||||||
|
declare -i total_selected_unverified=0
|
||||||
declare -i total_prevalidation_failed=0
|
declare -i total_prevalidation_failed=0
|
||||||
declare -i total_ok=0
|
declare -i total_ok=0
|
||||||
|
declare -i total_ok_unverified=0
|
||||||
declare -i total_write_errors=0
|
declare -i total_write_errors=0
|
||||||
declare -i total_failed_users=0
|
declare -i total_failed_users=0
|
||||||
|
|
||||||
@@ -1378,18 +1673,24 @@ for _user in "${selected_user_arr[@]}" ; do
|
|||||||
user_recovery_dir="${recovery_dir}/${_user}"
|
user_recovery_dir="${recovery_dir}/${_user}"
|
||||||
|
|
||||||
declare -i _user_selected=0
|
declare -i _user_selected=0
|
||||||
|
declare -i _user_selected_unverified=0
|
||||||
declare -i _user_prevalidation_failed=0
|
declare -i _user_prevalidation_failed=0
|
||||||
declare -a _user_prevalidation_failed_lines=()
|
declare -a _user_prevalidation_failed_lines=()
|
||||||
|
declare -A _origin_of_path=()
|
||||||
|
|
||||||
list_file="${LOCK_DIR}/list_${_user}.tsv"
|
list_file="${LOCK_DIR}/list_${_user}.tsv"
|
||||||
> "$list_file"
|
> "$list_file"
|
||||||
|
|
||||||
# =============
|
# =============
|
||||||
# --- Re-validate every VALID-marked entry against the CURRENT
|
# --- Re-validate every VALID- or plausible-UNVERIFIED-marked entry
|
||||||
# --- checks right before writing anything, and (unless dry-run)
|
# --- against the CURRENT checks right before writing anything, and
|
||||||
# --- back up the current on-disk ciphertext byte-for-byte first.
|
# --- (unless dry-run) back up the current on-disk ciphertext
|
||||||
|
# --- byte-for-byte first. original_size (from the recovery report,
|
||||||
|
# --- $3 of valid_entries_file) is passed through so the size-ratio
|
||||||
|
# --- part of the UNVERIFIED check can run for real here too, not
|
||||||
|
# --- just be trusted from the old report.
|
||||||
# =============
|
# =============
|
||||||
while IFS=$'\t' read -r _u _path ; do
|
while IFS=$'\t' read -r _u _path _origsize _recorded_class ; do
|
||||||
[[ "$_u" != "$_user" ]] && continue
|
[[ "$_u" != "$_user" ]] && continue
|
||||||
(( _user_selected++ ))
|
(( _user_selected++ ))
|
||||||
|
|
||||||
@@ -1402,13 +1703,17 @@ for _user in "${selected_user_arr[@]}" ; do
|
|||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
|
|
||||||
_val_result="$(validate_recovered_file "$_local" "$_path" "")"
|
_val_result="$(validate_recovered_file "$_local" "$_path" "$_origsize")"
|
||||||
_val_class="${_val_result%%|*}"
|
_val_class="${_val_result%%|*}"
|
||||||
_val_detail="${_val_result#*|}"
|
_val_detail="${_val_result#*|}"
|
||||||
|
|
||||||
if [[ "$_val_class" != "VALID" ]] ; then
|
if [[ "$_val_class" == "VALID" ]] ; then
|
||||||
|
: # dedicated structural check confirmed it - restore as usual
|
||||||
|
elif [[ "$_val_class" == "UNVERIFIED" && "$_val_detail" != *"LOOKS OFF"* ]] ; then
|
||||||
|
(( _user_selected_unverified++ ))
|
||||||
|
else
|
||||||
(( _user_prevalidation_failed++ ))
|
(( _user_prevalidation_failed++ ))
|
||||||
_user_prevalidation_failed_lines+=("${_path}"$'\t'"no longer validates as VALID (${_val_class}: ${_val_detail}) - skipped, not restored")
|
_user_prevalidation_failed_lines+=("${_path}"$'\t'"no longer validates as restorable (${_val_class}: ${_val_detail}) - skipped, not restored")
|
||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
|
|
||||||
@@ -1421,13 +1726,14 @@ for _user in "${selected_user_arr[@]}" ; do
|
|||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
_origin_of_path["$_path"]="$_val_class"
|
||||||
printf '%s\t%s\n' "$_path" "$_local" >> "$list_file"
|
printf '%s\t%s\n' "$_path" "$_local" >> "$list_file"
|
||||||
|
|
||||||
done < "$valid_entries_file"
|
done < "$valid_entries_file"
|
||||||
|
|
||||||
if [[ $_user_selected -eq 0 ]] ; then
|
if [[ $_user_selected -eq 0 ]] ; then
|
||||||
warn "No 'VALID' entries for account '${_user}' found in the selected report - skipping."
|
warn "No restorable (VALID or plausible UNVERIFIED) entries for account '${_user}' found in the selected report - skipping."
|
||||||
echo " [ keine VALID-Eintraege im gewaehlten Report ]" >> "$restore_report_file"
|
echo " [ keine VALID/UNVERIFIED-Eintraege im gewaehlten Report ]" >> "$restore_report_file"
|
||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
|
|
||||||
@@ -1440,10 +1746,15 @@ for _user in "${selected_user_arr[@]}" ; do
|
|||||||
fi
|
fi
|
||||||
|
|
||||||
if [[ -s "$list_file" ]] ; then
|
if [[ -s "$list_file" ]] ; then
|
||||||
|
# -d max_execution_time=0: re-validating and writing back every
|
||||||
|
# selected file of a large account can run well over an hour;
|
||||||
|
# without this the PHP CLI process is killed by PHP's own
|
||||||
|
# execution-time limit (typically inherited from the webserver's
|
||||||
|
# php.ini) mid-run.
|
||||||
if $DRY_RUN ; then
|
if $DRY_RUN ; then
|
||||||
su -c "$PHP_BIN $restore_php_file $_user $list_file --dry-run" -s /bin/bash $HTTP_USER > "$user_result_tsv" 2> "$log_file"
|
su -c "$PHP_BIN -d max_execution_time=0 $restore_php_file $_user $list_file --dry-run" -s /bin/bash $HTTP_USER > "$user_result_tsv" 2> "$log_file"
|
||||||
else
|
else
|
||||||
su -c "$PHP_BIN $restore_php_file $_user $list_file" -s /bin/bash $HTTP_USER > "$user_result_tsv" 2> "$log_file"
|
su -c "$PHP_BIN -d max_execution_time=0 $restore_php_file $_user $list_file" -s /bin/bash $HTTP_USER > "$user_result_tsv" 2> "$log_file"
|
||||||
fi
|
fi
|
||||||
_rc=$?
|
_rc=$?
|
||||||
else
|
else
|
||||||
@@ -1463,6 +1774,7 @@ for _user in "${selected_user_arr[@]}" ; do
|
|||||||
$terminal && echo ""
|
$terminal && echo ""
|
||||||
|
|
||||||
declare -i _user_ok=0
|
declare -i _user_ok=0
|
||||||
|
declare -i _user_ok_unverified=0
|
||||||
declare -i _user_write_errors=0
|
declare -i _user_write_errors=0
|
||||||
declare -a _user_write_error_lines=()
|
declare -a _user_write_error_lines=()
|
||||||
|
|
||||||
@@ -1471,8 +1783,13 @@ for _user in "${selected_user_arr[@]}" ; do
|
|||||||
[[ "$_path" = "path" ]] && continue
|
[[ "$_path" = "path" ]] && continue
|
||||||
[[ -z "$_path" ]] && continue
|
[[ -z "$_path" ]] && continue
|
||||||
|
|
||||||
|
_origin="${_origin_of_path[$_path]:-VALID}"
|
||||||
|
|
||||||
case "$_status" in
|
case "$_status" in
|
||||||
OK|DRY_RUN) (( _user_ok++ )) ;;
|
OK|DRY_RUN)
|
||||||
|
(( _user_ok++ ))
|
||||||
|
[[ "$_origin" == "UNVERIFIED" ]] && (( _user_ok_unverified++ ))
|
||||||
|
;;
|
||||||
*) (( _user_write_errors++ )); _user_write_error_lines+=("${_path}"$'\t'"${_status}: ${_detail}") ;;
|
*) (( _user_write_errors++ )); _user_write_error_lines+=("${_path}"$'\t'"${_status}: ${_detail}") ;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
@@ -1481,7 +1798,7 @@ for _user in "${selected_user_arr[@]}" ; do
|
|||||||
# reinterpret those backslashes as escape sequences and silently
|
# reinterpret those backslashes as escape sequences and silently
|
||||||
# mangle/eat parts of the text. printf's %s never reinterprets its
|
# mangle/eat parts of the text. printf's %s never reinterprets its
|
||||||
# argument, only the literal \t/\n in the format string itself.
|
# argument, only the literal \t/\n in the format string itself.
|
||||||
printf '%s\t%s\t%s\t%s\n' "$_path" "$_bytes" "$_status" "$_detail" >> "$restore_report_file"
|
printf '%s\t%s\t%s\t%s\t%s\n' "$_path" "$_bytes" "$_status" "$_origin" "$_detail" >> "$restore_report_file"
|
||||||
|
|
||||||
done < "$user_result_tsv"
|
done < "$user_result_tsv"
|
||||||
|
|
||||||
@@ -1489,7 +1806,10 @@ for _user in "${selected_user_arr[@]}" ; do
|
|||||||
|
|
||||||
if $terminal ; then
|
if $terminal ; then
|
||||||
echo -e " \033[1;37mAccount ${_user}\033[m"
|
echo -e " \033[1;37mAccount ${_user}\033[m"
|
||||||
echo -e " Ausgewaehlt (validation=VALID im Report)..: ${_user_selected}"
|
echo -e " Ausgewaehlt (VALID + plausibel UNVERIFIED).: ${_user_selected}"
|
||||||
|
if [[ $_user_selected_unverified -gt 0 ]] ; then
|
||||||
|
echo -e " davon 'Nicht pruefbar'/UNVERIFIED.......: \033[33m${_user_selected_unverified}\033[m"
|
||||||
|
fi
|
||||||
if [[ $_user_prevalidation_failed -gt 0 ]] ; then
|
if [[ $_user_prevalidation_failed -gt 0 ]] ; then
|
||||||
echo -e " Vor dem Schreiben aussortiert..............: \033[33m${_user_prevalidation_failed}\033[m"
|
echo -e " Vor dem Schreiben aussortiert..............: \033[33m${_user_prevalidation_failed}\033[m"
|
||||||
fi
|
fi
|
||||||
@@ -1501,6 +1821,9 @@ for _user in "${selected_user_arr[@]}" ; do
|
|||||||
else
|
else
|
||||||
echo -e " Zurueckgeschrieben & verifiziert...........: ${_user_ok} (${_user_pct_ok} %)"
|
echo -e " Zurueckgeschrieben & verifiziert...........: ${_user_ok} (${_user_pct_ok} %)"
|
||||||
fi
|
fi
|
||||||
|
if [[ $_user_ok_unverified -gt 0 ]] ; then
|
||||||
|
echo -e " davon UNVERIFIED (bitte stichprobenartig pruefen): \033[33m${_user_ok_unverified}\033[m"
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
if [[ $_user_write_errors -gt 0 ]] ; then
|
if [[ $_user_write_errors -gt 0 ]] ; then
|
||||||
echo -e " Fehler beim Schreiben/Verifizieren.........: \033[1;31m${_user_write_errors}\033[m"
|
echo -e " Fehler beim Schreiben/Verifizieren.........: \033[1;31m${_user_write_errors}\033[m"
|
||||||
@@ -1536,23 +1859,27 @@ for _user in "${selected_user_arr[@]}" ; do
|
|||||||
{
|
{
|
||||||
echo ""
|
echo ""
|
||||||
echo " Account ${_user}"
|
echo " Account ${_user}"
|
||||||
echo " Ausgewaehlt (validation=VALID im Report)..: ${_user_selected}"
|
echo " Ausgewaehlt (VALID + plausibel UNVERIFIED).: ${_user_selected}"
|
||||||
|
[[ $_user_selected_unverified -gt 0 ]] && echo " davon 'Nicht pruefbar'/UNVERIFIED.......: ${_user_selected_unverified}"
|
||||||
[[ $_user_prevalidation_failed -gt 0 ]] && echo " Vor dem Schreiben aussortiert..............: ${_user_prevalidation_failed}"
|
[[ $_user_prevalidation_failed -gt 0 ]] && echo " Vor dem Schreiben aussortiert..............: ${_user_prevalidation_failed}"
|
||||||
if $DRY_RUN ; then
|
if $DRY_RUN ; then
|
||||||
echo " Wuerde geschrieben & verifiziert werden....: ${_user_ok} (${_user_pct_ok} %)"
|
echo " Wuerde geschrieben & verifiziert werden....: ${_user_ok} (${_user_pct_ok} %)"
|
||||||
else
|
else
|
||||||
echo " Zurueckgeschrieben & verifiziert...........: ${_user_ok} (${_user_pct_ok} %)"
|
echo " Zurueckgeschrieben & verifiziert...........: ${_user_ok} (${_user_pct_ok} %)"
|
||||||
fi
|
fi
|
||||||
|
[[ $_user_ok_unverified -gt 0 ]] && echo " davon UNVERIFIED (bitte pruefen).........: ${_user_ok_unverified}"
|
||||||
[[ $_user_write_errors -gt 0 ]] && echo " Fehler beim Schreiben/Verifizieren.........: ${_user_write_errors}"
|
[[ $_user_write_errors -gt 0 ]] && echo " Fehler beim Schreiben/Verifizieren.........: ${_user_write_errors}"
|
||||||
} >> "$restore_report_file"
|
} >> "$restore_report_file"
|
||||||
|
|
||||||
(( total_selected += _user_selected ))
|
(( total_selected += _user_selected ))
|
||||||
|
(( total_selected_unverified += _user_selected_unverified ))
|
||||||
(( total_prevalidation_failed += _user_prevalidation_failed ))
|
(( total_prevalidation_failed += _user_prevalidation_failed ))
|
||||||
(( total_ok += _user_ok ))
|
(( total_ok += _user_ok ))
|
||||||
|
(( total_ok_unverified += _user_ok_unverified ))
|
||||||
(( total_write_errors += _user_write_errors ))
|
(( total_write_errors += _user_write_errors ))
|
||||||
|
|
||||||
unset _user_valid _user_prevalidation_failed _user_prevalidation_failed_lines
|
unset _user_valid _user_selected_unverified _user_prevalidation_failed _user_prevalidation_failed_lines
|
||||||
unset _user_ok _user_write_errors _user_write_error_lines
|
unset _user_ok _user_ok_unverified _user_write_errors _user_write_error_lines _origin_of_path
|
||||||
|
|
||||||
done
|
done
|
||||||
|
|
||||||
@@ -1568,20 +1895,24 @@ total_pct_ok="$(calc_percent "$total_ok" "$total_selected")"
|
|||||||
echo ""
|
echo ""
|
||||||
echo "Ausgewaehlte Accounts.......................: ${#selected_user_arr[@]}"
|
echo "Ausgewaehlte Accounts.......................: ${#selected_user_arr[@]}"
|
||||||
[[ $total_failed_users -gt 0 ]] && echo "Accounts mit Restore-Fehler.................: ${total_failed_users}"
|
[[ $total_failed_users -gt 0 ]] && echo "Accounts mit Restore-Fehler.................: ${total_failed_users}"
|
||||||
echo "Ausgewaehlt (validation=VALID im Report)....: ${total_selected}"
|
echo "Ausgewaehlt (VALID + plausibel UNVERIFIED)..: ${total_selected}"
|
||||||
|
[[ $total_selected_unverified -gt 0 ]] && echo " davon 'Nicht pruefbar'/UNVERIFIED.........: ${total_selected_unverified}"
|
||||||
[[ $total_prevalidation_failed -gt 0 ]] && echo "Vor dem Schreiben aussortiert...............: ${total_prevalidation_failed}"
|
[[ $total_prevalidation_failed -gt 0 ]] && echo "Vor dem Schreiben aussortiert...............: ${total_prevalidation_failed}"
|
||||||
if $DRY_RUN ; then
|
if $DRY_RUN ; then
|
||||||
echo "Wuerde geschrieben & verifiziert werden.....: ${total_ok} (${total_pct_ok} %)"
|
echo "Wuerde geschrieben & verifiziert werden.....: ${total_ok} (${total_pct_ok} %)"
|
||||||
else
|
else
|
||||||
echo "Zurueckgeschrieben & verifiziert insgesamt...: ${total_ok} (${total_pct_ok} %)"
|
echo "Zurueckgeschrieben & verifiziert insgesamt...: ${total_ok} (${total_pct_ok} %)"
|
||||||
fi
|
fi
|
||||||
|
[[ $total_ok_unverified -gt 0 ]] && echo " davon UNVERIFIED (bitte pruefen)..........: ${total_ok_unverified}"
|
||||||
[[ $total_write_errors -gt 0 ]] && echo "Fehler beim Schreiben/Verifizieren insgesamt: ${total_write_errors}"
|
[[ $total_write_errors -gt 0 ]] && echo "Fehler beim Schreiben/Verifizieren insgesamt: ${total_write_errors}"
|
||||||
echo ""
|
echo ""
|
||||||
if $DRY_RUN ; then
|
if $DRY_RUN ; then
|
||||||
echo "DRY-RUN: es wurde nichts geschrieben, gesichert oder veraendert."
|
echo "DRY-RUN: es wurde nichts geschrieben, gesichert oder veraendert."
|
||||||
else
|
else
|
||||||
echo "Chiffretext-Sicherung (vor dem Ueberschreiben) liegt unter: ${backup_dir}"
|
echo "Chiffretext-Sicherung (vor dem Ueberschreiben) liegt unter: ${backup_dir}"
|
||||||
echo "Bitte Ergebnisse trotz Verifikation stichprobenartig pruefen."
|
echo "Bitte Ergebnisse trotz Verifikation stichprobenartig pruefen - das gilt"
|
||||||
|
echo "besonders fuer die oben als UNVERIFIED markierten Dateien: dort gibt es"
|
||||||
|
echo "keine dedizierte Strukturpruefung, nur ein plausibles Groessenverhaeltnis."
|
||||||
fi
|
fi
|
||||||
} >> "$restore_report_file"
|
} >> "$restore_report_file"
|
||||||
|
|
||||||
@@ -1592,7 +1923,8 @@ if $terminal ; then
|
|||||||
echo ""
|
echo ""
|
||||||
echo -e " Ausgewaehlte Accounts.......................: ${#selected_user_arr[@]}"
|
echo -e " Ausgewaehlte Accounts.......................: ${#selected_user_arr[@]}"
|
||||||
[[ $total_failed_users -gt 0 ]] && echo -e " Accounts mit Restore-Fehler.................: \033[1;31m${total_failed_users}\033[m"
|
[[ $total_failed_users -gt 0 ]] && echo -e " Accounts mit Restore-Fehler.................: \033[1;31m${total_failed_users}\033[m"
|
||||||
echo -e " Ausgewaehlt (validation=VALID im Report)....: ${total_selected}"
|
echo -e " Ausgewaehlt (VALID + plausibel UNVERIFIED)..: ${total_selected}"
|
||||||
|
[[ $total_selected_unverified -gt 0 ]] && echo -e " davon 'Nicht pruefbar'/UNVERIFIED.........: \033[33m${total_selected_unverified}\033[m"
|
||||||
[[ $total_prevalidation_failed -gt 0 ]] && echo -e " Vor dem Schreiben aussortiert...............: \033[33m${total_prevalidation_failed}\033[m"
|
[[ $total_prevalidation_failed -gt 0 ]] && echo -e " Vor dem Schreiben aussortiert...............: \033[33m${total_prevalidation_failed}\033[m"
|
||||||
if $DRY_RUN ; then
|
if $DRY_RUN ; then
|
||||||
echo -e " Wuerde geschrieben & verifiziert werden.....: \033[1;32m${total_ok} (${total_pct_ok} %)\033[m"
|
echo -e " Wuerde geschrieben & verifiziert werden.....: \033[1;32m${total_ok} (${total_pct_ok} %)\033[m"
|
||||||
@@ -1602,6 +1934,7 @@ if $terminal ; then
|
|||||||
else
|
else
|
||||||
echo -e " Zurueckgeschrieben & verifiziert insgesamt...: ${total_ok} (${total_pct_ok} %)"
|
echo -e " Zurueckgeschrieben & verifiziert insgesamt...: ${total_ok} (${total_pct_ok} %)"
|
||||||
fi
|
fi
|
||||||
|
[[ $total_ok_unverified -gt 0 ]] && echo -e " davon UNVERIFIED (bitte pruefen)..........: \033[33m${total_ok_unverified}\033[m"
|
||||||
fi
|
fi
|
||||||
[[ $total_write_errors -gt 0 ]] && echo -e " Fehler beim Schreiben/Verifizieren insgesamt: \033[1;31m${total_write_errors}\033[m"
|
[[ $total_write_errors -gt 0 ]] && echo -e " Fehler beim Schreiben/Verifizieren insgesamt: \033[1;31m${total_write_errors}\033[m"
|
||||||
echo ""
|
echo ""
|
||||||
|
|||||||
+26
-1
@@ -582,6 +582,14 @@ fwrite(STDERR, "DEBUG: vor require lib/base.php\n");
|
|||||||
require_once __DIR__ . '/lib/base.php';
|
require_once __DIR__ . '/lib/base.php';
|
||||||
fwrite(STDERR, "DEBUG: nach require lib/base.php - Bootstrap abgeschlossen\n");
|
fwrite(STDERR, "DEBUG: nach require lib/base.php - Bootstrap abgeschlossen\n");
|
||||||
|
|
||||||
|
// Nextcloud's own bootstrap (lib/base.php) unconditionally calls
|
||||||
|
// set_time_limit(3600) as part of require_once above - this OVERRIDES
|
||||||
|
// whatever -d max_execution_time was passed on the PHP command line,
|
||||||
|
// since it is a later, explicit runtime call. Undo that here, now that
|
||||||
|
// the require is done, so a full-content scan of a large account isn't
|
||||||
|
// killed after exactly one hour regardless of the CLI flag.
|
||||||
|
set_time_limit(0);
|
||||||
|
|
||||||
chdir($oldWorkingDir);
|
chdir($oldWorkingDir);
|
||||||
|
|
||||||
if (function_exists('posix_getuid') && posix_getuid() === 0) {
|
if (function_exists('posix_getuid') && posix_getuid() === 0) {
|
||||||
@@ -771,6 +779,18 @@ if $terminal ; then
|
|||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
if $terminal ; then
|
||||||
|
echo ""
|
||||||
|
echo -e " \033[1mTemporäre Dateien – werden laufend aktualisiert:\033[m"
|
||||||
|
echo ""
|
||||||
|
echo -e " Fortschritt / Debug-Meldungen (PHP-Ausgabe auf STDERR):"
|
||||||
|
echo -e " \033[1mtail -f ${log_file}\033[m"
|
||||||
|
echo ""
|
||||||
|
echo -e " Rohergebnis pro Account (PHP-Ausgabe auf STDOUT, Datei für Datei):"
|
||||||
|
echo -e " \033[1mtail -f ${LOCK_DIR}/scan_<account>.tsv\033[m"
|
||||||
|
echo ""
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
||||||
# -----
|
# -----
|
||||||
# - Main part of the script
|
# - Main part of the script
|
||||||
@@ -810,7 +830,12 @@ for _user in "${selected_user_arr[@]}" ; do
|
|||||||
} >> "$report_file"
|
} >> "$report_file"
|
||||||
|
|
||||||
echononl " Scanning account \033[1;37m${_user}\033[m.."
|
echononl " Scanning account \033[1;37m${_user}\033[m.."
|
||||||
su -c "$PHP_BIN $scan_php_file $_user" -s /bin/bash $HTTP_USER > "$user_tsv" 2> "$log_file"
|
# -d max_execution_time=0: a full-content read of every file of a
|
||||||
|
# large account can easily run well over an hour; without this the
|
||||||
|
# PHP CLI process is killed by PHP's own execution-time limit
|
||||||
|
# (typically inherited from the webserver's php.ini) mid-scan, long
|
||||||
|
# before any actual error in the account's files.
|
||||||
|
su -c "$PHP_BIN -d max_execution_time=0 $scan_php_file $_user" -s /bin/bash $HTTP_USER > "$user_tsv" 2> "$log_file"
|
||||||
_rc=$?
|
_rc=$?
|
||||||
|
|
||||||
if [[ $_rc -ne 0 ]]; then
|
if [[ $_rc -ne 0 ]]; then
|
||||||
|
|||||||
Reference in New Issue
Block a user