25 Commits

Author SHA1 Message Date
ecenshu f501fd8671 Fix build warnings
ci / build_linux (push) Successful in 11m22s
ci / build_linux (pull_request) Successful in 5m26s
Snapshot now persisted in lastmodified date descending, hopefully aligns with snapshot simple check against first entry in a directory not existing in the snapshot during build
Building Snapshot should only be done synchrnonously and atomically
Blacklist watched for changes
2025-11-16 11:42:14 +10:30
ecenshu 8751a72176 feature/entry_from_filename (#4)
ci / build_linux (push) Failing after 13m35s
Build & Push Docker image / build-and-push (push) Failing after 13m40s
Reviewed-on: #4
Co-authored-by: Huy Nguyen <ecenshu@gmail.com>
Co-committed-by: Huy Nguyen <ecenshu@gmail.com>
2025-11-15 08:55:27 +00:00
ecenshu c260ebd566 If filename can extract to a NcaMetadata entry, don't use nspextractor to pull information (#3)
Build & Push Docker image / build-and-push (push) Successful in 5m39s
ci / build_linux (push) Successful in 4m36s
Scan directories sequentially to reduce memory footprint

Reviewed-on: #3
Co-authored-by: Huy Nguyen <ecenshu@gmail.com>
Co-committed-by: Huy Nguyen <ecenshu@gmail.com>
2025-11-15 06:59:25 +00:00
ecenshu 17d12fef6a Use a resource to initialise appsettings.json in config folder (#2)
Build & Push Docker image / build-and-push (push) Successful in 8m18s
ci / build_linux (push) Successful in 3m36s
Watch for KeySet, initial install will not have a valid value
TitleDatabase will use data folder

Reviewed-on: #2
Co-authored-by: Huy Nguyen <ecenshu@gmail.com>
Co-committed-by: Huy Nguyen <ecenshu@gmail.com>
2025-11-13 23:52:34 +00:00
ecenshu 97fc2dc872 Use sanitized github.ref
Build & Push Docker image / build-and-push (push) Successful in 15m11s
ci / build_linux (push) Successful in 5m29s
2025-11-13 19:44:36 +10:30
ecenshu a1ea34bc01 feature/ci (#1)
Build & Push Docker image / build-and-push (push) Has been cancelled
ci / build_linux (push) Has been cancelled
Consolidate data and config into separate folders that will be expected to be mapped in the container

Reviewed-on: #1
Co-authored-by: Huy Nguyen <ecenshu@gmail.com>
Co-committed-by: Huy Nguyen <ecenshu@gmail.com>
2025-11-13 09:11:21 +00:00
ecenshu 314af37b3d Workflow cache
Build & Push Docker image / build-and-push (push) Successful in 12m21s
ci / build_linux (push) Successful in 5m42s
2025-11-08 10:23:46 +10:30
ecenshu 877db194c2 tabs
Build & Push Docker image / build-and-push (push) Has been cancelled
ci / build_linux (push) Has been cancelled
2025-11-08 10:22:23 +10:30
ecenshu c4c15dbada workflow cache
Build & Push Docker image / build-and-push (push) Has been cancelled
2025-11-08 10:19:58 +10:30
ecenshu 301525e198 dots...
Build & Push Docker image / build-and-push (push) Waiting to run
ci / build_linux (push) Has been cancelled
2025-11-07 23:14:43 +10:30
ecenshu fa8f0d7443 LibHac.dll from Dependencies
Build & Push Docker image / build-and-push (push) Failing after 5m38s
ci / build_linux (push) Has been cancelled
2025-11-07 22:47:48 +10:30
ecenshu 1fdef1bcc8 Merge branch 'feature/gpt-oss-01'
Build & Push Docker image / build-and-push (push) Failing after 6m59s
ci / build_linux (push) Failing after 7m44s
2025-11-07 22:33:10 +10:30
ecenshu 53ba636258 At end maybe?
Build & Push Docker image / build-and-push (push) Failing after 5m14s
ci / build_linux (push) Has been cancelled
2025-11-07 22:23:45 +10:30
ecenshu a25f5f602e Hardcodepath
Build & Push Docker image / build-and-push (push) Failing after 1m50s
ci / build_linux (push) Has been cancelled
2025-11-07 22:17:31 +10:30
ecenshu b9370eb2d5 pathing problem?
Build & Push Docker image / build-and-push (push) Failing after 1m49s
ci / build_linux (push) Failing after 1m38s
2025-11-07 22:06:25 +10:30
ecenshu 7c6fba9b3f More copy pasta
Build & Push Docker image / build-and-push (push) Failing after 1m50s
ci / build_linux (push) Failing after 1m56s
2025-11-07 21:57:44 +10:30
ecenshu 4eb8324056 Vars instead of env
Build & Push Docker image / build-and-push (push) Failing after 1m49s
ci / build_linux (push) Failing after 1m34s
2025-11-07 21:50:27 +10:30
ecenshu 6cb78c91fa Use ENV for tags instead of SECRETS
Build & Push Docker image / build-and-push (push) Failing after 1m54s
ci / build_linux (push) Failing after 1m43s
2025-11-07 21:46:04 +10:30
ecenshu a9184acd23 Shell to lowercase
Build & Push Docker image / build-and-push (push) Failing after 1m57s
ci / build_linux (push) Failing after 1m32s
2025-11-07 21:40:17 +10:30
ecenshu fae1979e04 Tabs
Build & Push Docker image / build-and-push (push) Failing after 40s
ci / build_linux (push) Failing after 1m13s
2025-11-07 21:31:19 +10:30
ecenshu 36f2f0c2f9 BuildX doesn't like camelcase tags 2025-11-07 21:24:12 +10:30
ecenshu 35d4eccdfd Remove demo
Build & Push Docker image / build-and-push (push) Failing after 1m46s
ci / build_linux (push) Failing after 1m32s
Correct the tags to align with Gitea Repo expected tags
2025-11-07 21:19:38 +10:30
ecenshu a81f67536f ci copy pasta error
Gitea Actions Demo / Explore-Gitea-Actions (push) Has been cancelled
Build & Push Docker image / build-and-push (push) Failing after 1m55s
ci / build_linux (push) Failing after 1m32s
2025-11-07 20:50:20 +10:30
ecenshu ce68860175 First attempt with workflows
ci / tsan (C++17, tsan) (push) Waiting to run
ci / ubsan (C++17, ubsan) (push) Waiting to run
ci / results (push) Blocked by required conditions
Build & Push Docker image / build-and-push (push) Has been cancelled
ci / g++11.3.0 (C++17, Debug) (push) Has been cancelled
ci / g++12.3.0 (C++20, Debug) (push) Has been cancelled
ci / g++13.3.0 (C++20, Debug) (push) Has been cancelled
ci / g++14.3.0 (C++20, Debug) (push) Has been cancelled
ci / g++9.4.0 (C++17, Debug) (push) Has been cancelled
Gitea Actions Demo / Explore-Gitea-Actions (push) Has been cancelled
ci / clang20 (C++17, Debug) (push) Failing after 3m29s
ci / g++10.3.0 (C++17, Debug) (push) Has been cancelled
ci / clang15 (C++17, Debug) (push) Failing after 54s
ci / clang17 (C++17, Debug) (push) Failing after 54s
ci / clang18 (C++17, Debug) (push) Failing after 52s
ci / clang19 (C++17, Debug) (push) Failing after 53s
ci / g++15 (C++17, Debug) (push) Failing after 50s
ci / clang20 (C++17, Release) (push) Failing after 52s
ci / g++15 (C++17, Release) (push) Failing after 51s
ci / asan (C++17, asan) (push) Failing after 50s
2025-11-07 20:43:12 +10:30
ecenshu becc41a5f0 Testing actions
Gitea Actions Demo / Explore-Gitea-Actions (push) Successful in 2m29s
2025-11-04 08:35:55 +10:30
47 changed files with 1188 additions and 542 deletions
+1
View File
@@ -23,3 +23,4 @@
**/values.dev.yaml **/values.dev.yaml
LICENSE LICENSE
README.md README.md
data/
+136
View File
@@ -0,0 +1,136 @@
name: Build & Push Docker image
on:
# Trigger on pushes to main or release branches, and on manual workflow dispatch
push:
branches:
- main
- 'release/**'
- 'beta/**'
workflow_dispatch:
jobs:
build-and-push:
runs-on: ubuntu-latest
environment: production
steps:
- name: Convert to lowercase
id: github_repository_to_lowercase
run: |
# Grab the value (you can also use `${{ github.ref }}`, `${{ secrets.MY_SECRET }}`, etc.)
raw_value="${{ github.repository }}"
# Convert to lower case
lower_value=$(echo "$raw_value" | tr '[:upper:]' '[:lower:]')
# Export it to the workflow environment
# echo "MY_LOWER=$lower_value" >> $GITHUB_ENV
# If you want to use it as an output of this step:
echo "lowercase=$lower_value" >> $GITHUB_OUTPUT
- name: Convert ref to buildx safe value
id: docker_tag_from_ref
shell: bash
run: |
# Grab the raw ref
REF="${{ github.ref }}"
# Strip the "refs/*/" prefix (refs/heads/, refs/tags/…)
TAG=${REF#refs/*/}
# Replace characters that Docker tags disallow
# * "/" → "-"
# * ":" → "-"
# * Any other nonalphanumeric / . / _ / - → "-"
TAG=${TAG//\//-}
TAG=${TAG//:/-}
TAG=${TAG//[^a-zA-Z0-9._-]/-}
# (Optional) force lowercase Docker tags are casesensitive,
# but many people prefer lowercase
TAG=${TAG,,}
# Export to the action's output
echo "docker-tag=${TAG}" >> $GITHUB_OUTPUT
# ------------------------------------------------------------------
# 1. Checkout repository
# ------------------------------------------------------------------
- name: Checkout source
uses: actions/checkout@v4
with:
fetch-depth: 0 # needed for git revparse and tag generation
# ------------------------------------------------------------------
# 2. Set up Docker Buildx (optional, but recommended for multiarch)
# ------------------------------------------------------------------
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
# ------------------------------------------------------------------
# 3. Log in to the Gitea container registry
# ------------------------------------------------------------------
- name: Log in to Gitea registry
uses: docker/login-action@v3
with:
registry: ${{ secrets.REGISTRY_HOST }} # e.g. registry.example.com
username: ${{ secrets.REGISTRY_USER }} # e.g. admin
password: ${{ secrets.REGISTRY_PASSWORD }} # e.g. <apitoken>
- name: Cache Docker layers
uses: actions/cache@v3
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-
# ------------------------------------------------------------------
# 4. Build the Docker image
# ------------------------------------------------------------------
- name: Build image
id: build
uses: docker/build-push-action@v5
with:
context: .
file: TinfoilVibeServer/Dockerfile
# do not push yet
push: false
tags: |
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ github.sha }}
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ steps.docker_tag_from_ref.outputs.docker-tag }}
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:latest
build-args: |
# Add any build args here
# ARG_NAME=VALUE
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache
# ------------------------------------------------------------------
# 5. Push the image to the registry
# ------------------------------------------------------------------
- name: Push image
uses: docker/build-push-action@v5
with:
context: .
file: TinfoilVibeServer/Dockerfile
push: true
tags: |
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ github.sha }}
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ steps.docker_tag_from_ref.outputs.docker-tag }}
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:latest
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache
# ------------------------------------------------------------------
# 6. (Optional) Clean up local Docker cache
# ------------------------------------------------------------------
- name: Docker system prune
run: docker system prune -f
if: ${{ always() }}
# ------------------------------------------------------------------
# 7. Output useful info
# ------------------------------------------------------------------
- name: Show pushed image tags
run: |
echo "Pushed image tags:"
echo "- ${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ github.sha }}"
echo "- ${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ steps.docker_tag_from_ref.outputs.docker-tag }}"
echo "- ${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:latest"
+74
View File
@@ -0,0 +1,74 @@
name: ci
on: [push, pull_request]
jobs:
build_linux:
runs-on: ubuntu-latest
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Convert to lowercase
id: github_repository_to_lowercase
run: |
# Grab the value (you can also use `${{ github.ref }}`, `${{ secrets.MY_SECRET }}`, etc.)
raw_value="${{ github.repository }}"
# Convert to lower case
lower_value=$(echo "$raw_value" | tr '[:upper:]' '[:lower:]')
# Export it to the workflow environment
# echo "MY_LOWER=$lower_value" >> $GITHUB_ENV
# If you want to use it as an output of this step:
echo "lowercase=$lower_value" >> $GITHUB_OUTPUT
- name: Convert ref to buildx safe value
id: docker_tag_from_ref
shell: bash
run: |
# Grab the raw ref
REF="${{ github.ref }}"
# Strip the "refs/*/" prefix (refs/heads/, refs/tags/…)
TAG=${REF#refs/*/}
# Replace characters that Docker tags disallow
# * "/" → "-"
# * ":" → "-"
# * Any other nonalphanumeric / . / _ / - → "-"
TAG=${TAG//\//-}
TAG=${TAG//:/-}
TAG=${TAG//[^a-zA-Z0-9._-]/-}
# (Optional) force lowercase Docker tags are casesensitive,
# but many people prefer lowercase
TAG=${TAG,,}
# Export to the action's output
echo "docker-tag=${TAG}" >> $GITHUB_OUTPUT
- name: Cache Docker layers
uses: actions/cache@v3
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-
- name: Build image
id: build
uses: docker/build-push-action@v5
with:
context: .
file: TinfoilVibeServer/Dockerfile
# do not push yet
push: false
tags: |
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ github.sha }}
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ steps.docker_tag_from_ref.outputs.docker-tag }}
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:latest
build-args: |
# Add any build args here
# ARG_NAME=VALUE
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache
+7
View File
@@ -3,3 +3,10 @@ obj/
/packages/ /packages/
riderModule.iml riderModule.iml
/_ReSharper.Caches/ /_ReSharper.Caches/
TinfoilVibeServer.sln.DotSettings.user
data/*
!.data/.gitkeep
**/*.local.json
TinfoilVibeServer/config/prod.keys
TinfoilVibeServer/data/*
!TinfoilVibeServer/data/.gitkeep
+18
View File
@@ -0,0 +1,18 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="TinfoilVibeServer: http" type="LaunchSettings" factoryName=".NET Launch Settings Profile">
<option name="LAUNCH_PROFILE_PROJECT_FILE_PATH" value="$PROJECT_DIR$/TinfoilVibeServer/TinfoilVibeServer.csproj" />
<option name="LAUNCH_PROFILE_TFM" value="net9.0" />
<option name="LAUNCH_PROFILE_NAME" value="http" />
<option name="USE_EXTERNAL_CONSOLE" value="0" />
<option name="USE_MONO" value="0" />
<option name="RUNTIME_ARGUMENTS" value="" />
<option name="GENERATE_APPLICATIONHOST_CONFIG" value="1" />
<option name="SHOW_IIS_EXPRESS_OUTPUT" value="0" />
<option name="SEND_DEBUG_REQUEST" value="1" />
<option name="ADDITIONAL_IIS_EXPRESS_ARGUMENTS" value="" />
<option name="AUTO_ATTACH_CHILDREN" value="0" />
<method v="2">
<option name="Build" />
</method>
</configuration>
</component>
+54
View File
@@ -0,0 +1,54 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="TinfoilVibeServer/Dockerfile" type="docker-deploy" factoryName="dockerfile" server-name="Docker WSL">
<deployment type="dockerfile">
<settings>
<option name="imageTag" value="gitea.ecenshu.net/ecenshu/tinfoilvibeserver:dev" />
<option name="containerName" value="tinfoilvibeserver" />
<option name="contextFolderPath" value="." />
<option name="envVars">
<list>
<DockerEnvVarImpl>
<option name="name" value="uid" />
<option name="value" value="1034" />
</DockerEnvVarImpl>
</list>
</option>
<option name="portBindings">
<list>
<DockerPortBindingImpl>
<option name="containerPort" value="8080" />
<option name="hostIp" value="127.0.0.1" />
<option name="hostPort" value="8080" />
</DockerPortBindingImpl>
</list>
</option>
<option name="showCommandPreview" value="true" />
<option name="sourceFilePath" value="TinfoilVibeServer/Dockerfile" />
<option name="volumeBindings">
<list>
<DockerVolumeBindingImpl>
<option name="containerPath" value="/app/data" />
<option name="hostPath" value="D:\Cloud\Git\TinfoilVibeServer\TinfoilVibeServer\bin\Debug\net9.0\data" />
</DockerVolumeBindingImpl>
<DockerVolumeBindingImpl>
<option name="containerPath" value="/app/config" />
<option name="hostPath" value="D:\Cloud\Git\TinfoilVibeServer\TinfoilVibeServer\bin\Debug\net9.0\config" />
</DockerVolumeBindingImpl>
<DockerVolumeBindingImpl>
<option name="containerPath" value="/roms_cold" />
<option name="hostPath" value="Z:\downloads\roms\switch" />
<option name="readOnly" value="true" />
</DockerVolumeBindingImpl>
<DockerVolumeBindingImpl>
<option name="containerPath" value="/roms_hot" />
<option name="hostPath" value="Z:\imgs\roms\Switch" />
<option name="readOnly" value="true" />
</DockerVolumeBindingImpl>
</list>
</option>
</settings>
</deployment>
<EXTENSION ID="com.jetbrains.rider.docker.debug" isFastModeEnabled="true" isSslEnabled="false" />
<method v="2" />
</configuration>
</component>
+1
View File
@@ -3,6 +3,7 @@ Microsoft Visual Studio Solution File, Format Version 12.00
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{41999D26-405C-4DAB-8991-CBA992117C84}" Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{41999D26-405C-4DAB-8991-CBA992117C84}"
ProjectSection(SolutionItems) = preProject ProjectSection(SolutionItems) = preProject
compose.yaml = compose.yaml compose.yaml = compose.yaml
compose.overide.yaml = compose.overide.yaml
EndProjectSection EndProjectSection
EndProject EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TinfoilVibeServer", "TinfoilVibeServer\TinfoilVibeServer.csproj", "{DE992FDB-6D13-4152-925D-29D39A23FB75}" Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TinfoilVibeServer", "TinfoilVibeServer\TinfoilVibeServer.csproj", "{DE992FDB-6D13-4152-925D-29D39A23FB75}"
+2 -1
View File
@@ -1,4 +1,5 @@
<wpf:ResourceDictionary xml:space="preserve" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:s="clr-namespace:System;assembly=mscorlib" xmlns:ss="urn:shemas-jetbrains-com:settings-storage-xaml" xmlns:wpf="http://schemas.microsoft.com/winfx/2006/xaml/presentation"> <wpf:ResourceDictionary xml:space="preserve" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:s="clr-namespace:System;assembly=mscorlib" xmlns:ss="urn:shemas-jetbrains-com:settings-storage-xaml" xmlns:wpf="http://schemas.microsoft.com/winfx/2006/xaml/presentation">
<s:String x:Key="/Default/CodeStyle/Naming/CSharpNaming/Abbreviations/=IP/@EntryIndexedValue">IP</s:String> <s:String x:Key="/Default/CodeStyle/Naming/CSharpNaming/Abbreviations/=IP/@EntryIndexedValue">IP</s:String>
<s:String x:Key="/Default/CodeStyle/Naming/CSharpNaming/Abbreviations/=NSP/@EntryIndexedValue">NSP</s:String> <s:String x:Key="/Default/CodeStyle/Naming/CSharpNaming/Abbreviations/=NSP/@EntryIndexedValue">NSP</s:String>
<s:String x:Key="/Default/CodeStyle/Naming/CSharpNaming/Abbreviations/=PFS/@EntryIndexedValue">PFS</s:String></wpf:ResourceDictionary> <s:String x:Key="/Default/CodeStyle/Naming/CSharpNaming/Abbreviations/=PFS/@EntryIndexedValue">PFS</s:String>
<s:String x:Key="/Default/CodeStyle/Naming/CSharpNaming/Abbreviations/=ROM/@EntryIndexedValue">ROM</s:String></wpf:ResourceDictionary>
+3
View File
@@ -87,7 +87,10 @@
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUniqueRef_00601_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FDecompilerCache_003Fdecompiler_003F77078e9a1d254191bb508f54a277fc6e1c2e00_003Fa0_003F631946a0_003FUniqueRef_00601_002Ecs/@EntryIndexedValue">ForceIncluded</s:String> <s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUniqueRef_00601_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FDecompilerCache_003Fdecompiler_003F77078e9a1d254191bb508f54a277fc6e1c2e00_003Fa0_003F631946a0_003FUniqueRef_00601_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUnpack_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FSourcesCache_003F4c375bcb6a2d2378855cad1b1c7cfe7ca1448866f1e8af44226775b5f75df86_003FUnpack_002Ecs/@EntryIndexedValue">ForceIncluded</s:String> <s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUnpack_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FSourcesCache_003F4c375bcb6a2d2378855cad1b1c7cfe7ca1448866f1e8af44226775b5f75df86_003FUnpack_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUnsafeHelpers_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FDecompilerCache_003Fdecompiler_003F77078e9a1d254191bb508f54a277fc6e1c2e00_003Fd1_003Fc59f91c2_003FUnsafeHelpers_002Ecs/@EntryIndexedValue">ForceIncluded</s:String> <s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUnsafeHelpers_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FDecompilerCache_003Fdecompiler_003F77078e9a1d254191bb508f54a277fc6e1c2e00_003Fd1_003Fc59f91c2_003FUnsafeHelpers_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUriExt_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E3_003Fresharper_002Dhost_003FSourcesCache_003F25edf3e734d0c76e87b29a9954b8b4a7383648a69396554742e5529205e2dd7_003FUriExt_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUriSyntax_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E3_003Fresharper_002Dhost_003FSourcesCache_003F7ed82aa0f48a6bf284b4aba7c70aff35142349e44fb4f6caec3d71611f9929_003FUriSyntax_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUtility_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FSourcesCache_003F3ea3ed6216d2412ac7c33016ad940618bcfbcafe1633dc26832be514633b4_003FUtility_002Ecs/@EntryIndexedValue">ForceIncluded</s:String> <s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUtility_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FSourcesCache_003F3ea3ed6216d2412ac7c33016ad940618bcfbcafe1633dc26832be514633b4_003FUtility_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AWaitHandle_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E3_003Fresharper_002Dhost_003FSourcesCache_003Fcab54c7c1f8d9b3da5f7878e39faba52467f16d0899a2c4b10086cb2ef73f2b_003FWaitHandle_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AXciPartition_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FDecompilerCache_003Fdecompiler_003F77078e9a1d254191bb508f54a277fc6e1c2e00_003F04_003F4e8815da_003FXciPartition_002Ecs/@EntryIndexedValue">ForceIncluded</s:String> <s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AXciPartition_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FDecompilerCache_003Fdecompiler_003F77078e9a1d254191bb508f54a277fc6e1c2e00_003F04_003F4e8815da_003FXciPartition_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AZipArchive_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FSourcesCache_003F39ececcf144e1f9c884152723ed93931cd232485eaf2824bf5beb526f1f321b_003FZipArchive_002Ecs/@EntryIndexedValue">ForceIncluded</s:String> <s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AZipArchive_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FSourcesCache_003F39ececcf144e1f9c884152723ed93931cd232485eaf2824bf5beb526f1f321b_003FZipArchive_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
+104 -47
View File
@@ -12,7 +12,7 @@ public interface IAuthStore
void Dispose(); void Dispose();
bool TryValidate(string username, bool TryValidate(string username,
string password, string password,
int? uid, string? uid,
string ip, string ip,
out string? error); out string? error);
@@ -29,83 +29,106 @@ public class AuthStore : IDisposable, IAuthStore
{ {
private readonly ILogger<AuthStore> _logger; private readonly ILogger<AuthStore> _logger;
private readonly ConfigManager _configManager; private readonly ConfigManager _configManager;
private readonly IHostEnvironment _env;
public readonly ConcurrentDictionary<string, Credential> Credentials = new(); public readonly ConcurrentDictionary<string, Credential> Credentials = new();
public readonly ConcurrentDictionary<string, List<int>> Fingerprints = new(); public readonly ConcurrentDictionary<string, List<string>> Fingerprints = new();
public readonly ConcurrentDictionary<string, int> FailedAttempts = new(); public readonly ConcurrentDictionary<string, int> FailedAttempts = new();
private readonly HashSet<string> BlacklistIPs = new(); private readonly HashSet<string> _blacklistIPs = new();
private readonly object _sync = new(); private readonly Lock _sync = new();
private readonly FileSystemWatcher _credentialsWatcher; private readonly FileSystemWatcher _credentialsWatcher;
private readonly FileSystemWatcher _blacklistWatcher;
public AuthStore(ILogger<AuthStore> logger, ConfigManager configManager) public AuthStore(ILogger<AuthStore> logger, ConfigManager configManager, IHostEnvironment env)
{ {
_logger = logger; _logger = logger;
_configManager = configManager; _configManager = configManager;
_env = env;
LoadAll(); LoadAll();
var credentialsFilePath = DetermineCredentialsPath(_configManager.Settings?.CredentialsFile, env);
var directoryName = Path.GetDirectoryName(_configManager.Settings.CredentialsFile); var directoryName = Path.GetDirectoryName(Path.Combine(credentialsFilePath));
_credentialsWatcher = new FileSystemWatcher _credentialsWatcher = new FileSystemWatcher
{ {
Path = (!string.IsNullOrEmpty(directoryName)) ? directoryName : AppContext.BaseDirectory, Path = (!string.IsNullOrEmpty(directoryName)) ? directoryName : AppContext.BaseDirectory,
Filter = Path.GetFileName(_configManager.Settings.CredentialsFile), Filter = Path.GetFileName(credentialsFilePath) ?? throw new InvalidOperationException(),
NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.Size | NotifyFilters.Attributes NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.Size | NotifyFilters.Attributes
}; };
_credentialsWatcher.Changed += (_, _) => OnCredentialsChanged(); _credentialsWatcher.Changed += (_, _) => OnCredentialsChanged();
_credentialsWatcher.EnableRaisingEvents = true; _credentialsWatcher.EnableRaisingEvents = true;
_blacklistWatcher = new FileSystemWatcher
{
Path = (!string.IsNullOrEmpty(directoryName)) ? directoryName : AppContext.BaseDirectory,
Filter = Path.GetFileName(_configManager.Settings?.BlacklistFile) ?? throw new InvalidOperationException(),
NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.Size | NotifyFilters.Attributes
};
_blacklistWatcher.Changed += (_, _) => OnBlacklistChanged();
_blacklistWatcher.EnableRaisingEvents = true;
_logger.LogInformation("Started watching credentials file {File}", credentialsFilePath);
_logger.LogInformation("Started watching blacklist file {File}", Path.Combine(env.ContentRootPath, "data", Path.GetFileName(_configManager.Settings?.BlacklistFile) ?? throw new InvalidOperationException()));
}
private static string DetermineCredentialsPath(string? settingsCredentialsFile, IHostEnvironment env)
{
if (settingsCredentialsFile == null) return Path.Combine("app","data","credentials.json");
return Path.IsPathRooted(settingsCredentialsFile) ? settingsCredentialsFile : Path.Combine(env.ContentRootPath,"data",settingsCredentialsFile);
} }
public void Dispose() public void Dispose()
{ {
_credentialsWatcher?.Dispose(); _credentialsWatcher.Dispose();
} }
#region Loading helpers #region Loading helpers
private void LoadAll() private void LoadAll()
{ {
_logger.LogInformation("Loading authentication data from {File}", _configManager.Settings.CredentialsFile); var credentialsFilePath = DetermineCredentialsPath(_configManager.Settings?.CredentialsFile, _env);
_logger.LogInformation("Loading authentication data from {File}", credentialsFilePath);
// credentials // credentials
if (File.Exists(_configManager.Settings.CredentialsFile)) if (File.Exists(credentialsFilePath))
{ {
var txt = File.ReadAllText(_configManager.Settings.CredentialsFile); var txt = File.ReadAllText(credentialsFilePath);
var dict = JsonSerializer.Deserialize<Dictionary<string, Credential>>(txt)!; var dict = JsonSerializer.Deserialize<Dictionary<string, Credential>>(txt)!;
foreach (var kv in dict) foreach (var kv in dict)
Credentials[kv.Key] = kv.Value; Credentials[kv.Key] = kv.Value;
} }
else else
{ {
FileSystemExtensions.EnsureDirectoryExists(Path.GetDirectoryName(Path.GetFullPath(_configManager.Settings.CredentialsFile))); FileSystemExtensions.EnsureDirectoryExists(Path.GetDirectoryName(Path.GetFullPath(_configManager.Settings?.CredentialsFile ?? throw new InvalidOperationException())));
} }
// fingerprints // fingerprints
if (File.Exists(_configManager.Settings.FingerprintsFile)) if (File.Exists(_configManager.Settings?.FingerprintsFile))
{ {
var txt = File.ReadAllText(_configManager.Settings.FingerprintsFile); var txt = File.ReadAllText(_configManager.Settings.FingerprintsFile);
var dict = JsonSerializer.Deserialize<Dictionary<string, List<int>>>(txt)!; var dict = JsonSerializer.Deserialize<Dictionary<string, List<string>>>(txt)!;
foreach (var kv in dict) foreach (var kv in dict)
Fingerprints[kv.Key] = kv.Value; Fingerprints[kv.Key] = kv.Value;
} }
else else
{ {
if (_configManager.Settings?.FingerprintsFile != null)
FileSystemExtensions.EnsureDirectoryExists(Path.GetDirectoryName(Path.GetFullPath(_configManager.Settings.FingerprintsFile))); FileSystemExtensions.EnsureDirectoryExists(Path.GetDirectoryName(Path.GetFullPath(_configManager.Settings.FingerprintsFile)));
} }
// blacklist // blacklist
if (File.Exists(_configManager.Settings.BlacklistFile)) if (File.Exists(_configManager.Settings?.BlacklistFile))
{ {
var txt = File.ReadAllText(_configManager.Settings.BlacklistFile); var txt = File.ReadAllText(_configManager.Settings.BlacklistFile);
var arr = JsonSerializer.Deserialize<string[]>(txt)!; var arr = JsonSerializer.Deserialize<string[]>(txt)!;
foreach (var ip in arr) foreach (var ip in arr)
BlacklistIPs.Add(ip); _blacklistIPs.Add(ip);
} }
else else
{ {
if (_configManager.Settings?.BlacklistFile != null)
FileSystemExtensions.EnsureDirectoryExists(Path.GetDirectoryName(Path.GetFullPath(_configManager.Settings.BlacklistFile))); FileSystemExtensions.EnsureDirectoryExists(Path.GetDirectoryName(Path.GetFullPath(_configManager.Settings.BlacklistFile)));
} }
_logger.LogInformation("Loaded {UserCount} users, {FpCount} fingerprints, {IpCount} IPs", _logger.LogInformation("Loaded {UserCount} users, {FpCount} fingerprints, {IpCount} IPs",
Credentials.Count, Fingerprints.Count, BlacklistIPs.Count); Credentials.Count, Fingerprints.Count, _blacklistIPs.Count);
} }
#endregion #endregion
@@ -122,17 +145,28 @@ public class AuthStore : IDisposable, IAuthStore
}); });
} }
private void OnBlacklistChanged()
{
// Small debounce the file may still be locked by the editor.
Task.Run(async () =>
{
await Task.Delay(200);
LoadAll();
});
}
private void ReloadCredentials() private void ReloadCredentials()
{ {
if (!File.Exists(_configManager.Settings.CredentialsFile)) var credentialsFilePath = DetermineCredentialsPath(_configManager.Settings?.CredentialsFile, _env);
if (!File.Exists(credentialsFilePath))
{ {
_logger.LogError("Credentials file {File} does not exist", _configManager.Settings.CredentialsFile); _logger.LogError("Credentials file {File} does not exist", credentialsFilePath);
return; return;
} }
try try
{ {
var txt = File.ReadAllText(_configManager.Settings.CredentialsFile); var txt = File.ReadAllText(credentialsFilePath);
var newDict = JsonSerializer.Deserialize<Dictionary<string, Credential>>(txt)!; var newDict = JsonSerializer.Deserialize<Dictionary<string, Credential>>(txt)!;
lock (_sync) lock (_sync)
@@ -164,7 +198,7 @@ public class AuthStore : IDisposable, IAuthStore
} }
catch (Exception ex) catch (Exception ex)
{ {
_logger.LogError(ex, "Failed to reload credentials from {File}", _configManager.Settings.CredentialsFile); _logger.LogError(ex, "Failed to reload credentials from {File}", credentialsFilePath);
// ignore malformed JSON or IO error keep old state // ignore malformed JSON or IO error keep old state
} }
} }
@@ -173,11 +207,7 @@ public class AuthStore : IDisposable, IAuthStore
#region Authentication logic #region Authentication logic
public bool TryValidate(string username, public bool TryValidate(string username, string password, string? uid, string ip, out string? error)
string password,
int? uid,
string ip,
out string? error)
{ {
error = null; error = null;
lock (_sync) lock (_sync)
@@ -190,14 +220,6 @@ public class AuthStore : IDisposable, IAuthStore
PersistCredentials(); PersistCredentials();
_logger.LogInformation("Created new user {Username} (verified={Verified})", username, cred.Verified); _logger.LogInformation("Created new user {Username} (verified={Verified})", username, cred.Verified);
var list = Fingerprints.GetOrAdd(username, _ => new List<int>());
if (uid.HasValue && !list.Contains(uid.Value))
{
if (list.Count < cred.AllowedUidCount)
list.Add(uid.Value);
PersistFingerprints();
}
error = "User not verified"; error = "User not verified";
IncrementFailed(username, ip); IncrementFailed(username, ip);
return false; return false;
@@ -218,15 +240,15 @@ public class AuthStore : IDisposable, IAuthStore
return false; return false;
} }
if (uid.HasValue) if (uid != null)
{ {
var list = Fingerprints.GetOrAdd(username, _ => new List<int>()); var list = Fingerprints.GetOrAdd(username, _ => new List<string>());
if (!list.Contains(uid.Value)) if (!list.Contains(uid))
{ {
if (list.Count < cred.AllowedUidCount) if (list.Count < cred.AllowedUidCount)
{ {
list.Add(uid.Value); list.Add(uid);
PersistFingerprints(); PersistFingerprints();
} }
else else
@@ -237,6 +259,11 @@ public class AuthStore : IDisposable, IAuthStore
} }
} }
} }
else
{
_logger.LogWarning("No Fingerprint given during authentication for {Username} from {IP}", username, ip);
return false;
}
FailedAttempts[username] = 0; FailedAttempts[username] = 0;
return true; return true;
@@ -245,24 +272,31 @@ public class AuthStore : IDisposable, IAuthStore
public int IncrementFailed(string username, string ip) public int IncrementFailed(string username, string ip)
{ {
var newCount = FailedAttempts.GetOrAdd(username, 0) + 1;
lock (_sync) lock (_sync)
{ {
var newCount = FailedAttempts.GetOrAdd(username, 0) + 1;
FailedAttempts[username] = newCount; FailedAttempts[username] = newCount;
}
_logger.LogInformation("Failed attempts for {Username} increased to {Count}", username, newCount); _logger.LogInformation("Failed attempts for {Username} increased to {Count}", username, newCount);
if (_configManager.Settings == null)
{
_logger.LogCritical("Settings not set to determine failed login counts");
return int.MaxValue;
}
if (newCount < _configManager.Settings.MaxFailedAttempts + 1) return newCount; if (newCount < _configManager.Settings.MaxFailedAttempts + 1) return newCount;
BlacklistIPs.Add(ip); _blacklistIPs.Add(ip);
PersistBlacklist(); PersistBlacklist();
lock (_sync) lock (_sync)
{ {
FailedAttempts[username] = 0; FailedAttempts[username] = 0;
} }
_logger.LogWarning("IP {IP} blacklisted after {Count} failures", ip, newCount); _logger.LogWarning("IP {IP} blacklisted after {Count} failures", ip, newCount);
return newCount; return newCount;
} }
}
#endregion #endregion
@@ -280,38 +314,61 @@ public class AuthStore : IDisposable, IAuthStore
private void PersistCredentials() private void PersistCredentials()
{ {
var credentialsFilePath = DetermineCredentialsPath(_configManager.Settings?.CredentialsFile, _env);
var json = JsonSerializer.Serialize(Credentials, new JsonSerializerOptions { WriteIndented = true }); var json = JsonSerializer.Serialize(Credentials, new JsonSerializerOptions { WriteIndented = true });
File.WriteAllText(_configManager.Settings.CredentialsFile, json); if (_configManager.Settings == null)
{
_logger.LogCritical("Credentials file not set, cannot persist");
}
else
{
File.WriteAllText(credentialsFilePath, json);
}
} }
private void PersistFingerprints() private void PersistFingerprints()
{ {
var json = JsonSerializer.Serialize(Fingerprints, new JsonSerializerOptions { WriteIndented = true }); var json = JsonSerializer.Serialize(Fingerprints, new JsonSerializerOptions { WriteIndented = true });
if (_configManager.Settings == null)
{
_logger.LogCritical("Fingerprint file not set, cannot persist");
}
else
{
File.WriteAllText(_configManager.Settings.FingerprintsFile, json); File.WriteAllText(_configManager.Settings.FingerprintsFile, json);
} }
}
private void PersistBlacklist() private void PersistBlacklist()
{ {
var json = JsonSerializer.Serialize(BlacklistIPs.ToArray(), new JsonSerializerOptions { WriteIndented = true }); var json = JsonSerializer.Serialize(_blacklistIPs.ToArray(), new JsonSerializerOptions { WriteIndented = true });
if (_configManager.Settings == null)
{
_logger.LogCritical("Blacklist file not set, cannot persist");
}
else
{
File.WriteAllText(_configManager.Settings.BlacklistFile, json); File.WriteAllText(_configManager.Settings.BlacklistFile, json);
} }
}
#endregion #endregion
#region Blacklist helpers #region Blacklist helpers
public bool IsIPBlacklisted(string ipAddress) public bool IsIPBlacklisted(string ipAddress)
{ {
return BlacklistIPs.Contains(ipAddress); return _blacklistIPs.Contains(ipAddress);
} }
public bool UnbanIp(string ipAddress) public bool UnbanIp(string ipAddress)
{ {
return BlacklistIPs.Remove(ipAddress); return _blacklistIPs.Remove(ipAddress);
} }
public bool BlacklistActive() public bool BlacklistActive()
{ {
return BlacklistIPs.Count > 0; return _blacklistIPs.Count > 0;
} }
#endregion #endregion
@@ -16,7 +16,7 @@ public class CancelableFileResult : FileResult
/// Allows you to set a suggested download name. /// Allows you to set a suggested download name.
/// It will be sent in a “ContentDisposition” header. /// It will be sent in a “ContentDisposition” header.
/// </summary> /// </summary>
public string? FileDownloadName { get; set; } public new string? FileDownloadName { get; set; }
public override async Task ExecuteResultAsync(ActionContext context) public override async Task ExecuteResultAsync(ActionContext context)
{ {
@@ -1,33 +1,15 @@
using System; using Microsoft.AspNetCore.Mvc;
using System.IO;
using System.Linq;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;
using SharpCompress.Readers; using SharpCompress.Readers;
using TinfoilVibeServer.Models; using TinfoilVibeServer.Models;
using TinfoilVibeServer.Services; using TinfoilVibeServer.Services;
using TinfoilVibeServer.Utilities;
namespace TinfoilVibeServer.Controllers; namespace TinfoilVibeServer.Controllers;
[ApiController] [ApiController]
[Route("/")] [Route("/")]
public sealed class IndexController : ControllerBase public sealed class IndexController(ISnapshotService snapshotService, IndexBuilderService indexBuilderService) : ControllerBase
{ {
private readonly ISnapshotService _snapshotService;
private readonly TitleDatabaseService _titleDb;
private readonly IConfiguration _configuration;
private readonly IndexBuilderService _indexBuilderService;
public IndexController(ISnapshotService snapshotService,
TitleDatabaseService titleDb,
IConfiguration configuration, IndexBuilderService indexBuilderService)
{
_snapshotService = snapshotService;
_titleDb = titleDb;
_configuration = configuration;
_indexBuilderService = indexBuilderService;
}
// ------------------------------------------------------------ // ------------------------------------------------------------
// GET / // GET /
// ------------------------------------------------------------ // ------------------------------------------------------------
@@ -40,10 +22,10 @@ public sealed class IndexController : ControllerBase
{ {
if (HttpContext.Request.Headers.CacheControl == "no-cache") if (HttpContext.Request.Headers.CacheControl == "no-cache")
{ {
_indexBuilderService.InvalidateIndex(this, EventArgs.Empty); indexBuilderService.InvalidateIndex(this, EventArgs.Empty);
} }
var index = _indexBuilderService.Build(HttpContext); var index = indexBuilderService.Build(HttpContext);
return Ok(index); return Ok(index);
} }
@@ -75,7 +57,7 @@ public sealed class IndexController : ControllerBase
var titleId = match.Groups["id"].Value.ToUpperInvariant(); var titleId = match.Groups["id"].Value.ToUpperInvariant();
// ---- 2️⃣ Find the file that contains this TitleId ------------ // ---- 2️⃣ Find the file that contains this TitleId ------------
var entry = _snapshotService.GetSnapshot().Files var entry = snapshotService.GetSnapshot().Files
.FirstOrDefault(e => { return e.Titles.FirstOrDefault(hash => hash.TitleId == titleId)?.TitleId == titleId; }); .FirstOrDefault(e => { return e.Titles.FirstOrDefault(hash => hash.TitleId == titleId)?.TitleId == titleId; });
if (entry == null) if (entry == null)
@@ -84,7 +66,7 @@ public sealed class IndexController : ControllerBase
// ---- 3️⃣ If the file is a normal NSP → send it ---------------- // ---- 3️⃣ If the file is a normal NSP → send it ----------------
if (Path.GetExtension(entry.Path).Equals(".nsp", StringComparison.OrdinalIgnoreCase) if (Path.GetExtension(entry.Path).Equals(".nsp", StringComparison.OrdinalIgnoreCase)
&& !entry.Path.Contains(_snapshotService.GetArchivePathSeparator())) && !entry.Path.Contains(snapshotService.GetArchivePathSeparator()))
{ {
if (System.IO.File.Exists(entry.Path)) if (System.IO.File.Exists(entry.Path))
{ {
@@ -94,7 +76,7 @@ public sealed class IndexController : ControllerBase
FileMode.Open, FileMode.Open,
FileAccess.Read, FileAccess.Read,
FileShare.Read, FileShare.Read,
bufferSize: 128 * 1024 * 1024, // 81920, // 80KiB bufferSize: 128 * 1024 * 1024, // 81920, // 80 KiB
useAsync: true); // <‑‑ VERY important for scalability useAsync: true); // <‑‑ VERY important for scalability
// 2️⃣ Return a cancellationaware result // 2️⃣ Return a cancellationaware result
@@ -116,7 +98,7 @@ public sealed class IndexController : ControllerBase
if (IsInsideArchive(entry.Path)) if (IsInsideArchive(entry.Path))
{ {
// Example: file is inside an archive use ArchiveHandler // Example: file is inside an archive use ArchiveHandler
var innerFileName = entry.Path.Split(_snapshotService.GetArchivePathSeparator()).Last(); var innerFileName = entry.Path.Split(snapshotService.GetArchivePathSeparator()).Last();
var stream = StreamFromArchive(entry, titleId, out var streamContainer); var stream = StreamFromArchive(entry, titleId, out var streamContainer);
if (stream == null) if (stream == null)
@@ -149,7 +131,7 @@ public sealed class IndexController : ControllerBase
// (e.g. "Games/MyGame.nsp" is a regular file; "archive.7z/mygame.nsp" // (e.g. "Games/MyGame.nsp" is a regular file; "archive.7z/mygame.nsp"
// would be inside an archive). For simplicity we only check // would be inside an archive). For simplicity we only check
// for common archive extensions. // for common archive extensions.
var filePath = path.Split(_snapshotService.GetArchivePathSeparator()).First(); var filePath = path.Split(snapshotService.GetArchivePathSeparator()).First();
return filePath.EndsWith(".zip", StringComparison.OrdinalIgnoreCase) || return filePath.EndsWith(".zip", StringComparison.OrdinalIgnoreCase) ||
filePath.EndsWith(".7z", StringComparison.OrdinalIgnoreCase) || filePath.EndsWith(".7z", StringComparison.OrdinalIgnoreCase) ||
filePath.EndsWith(".rar", StringComparison.OrdinalIgnoreCase); filePath.EndsWith(".rar", StringComparison.OrdinalIgnoreCase);
@@ -163,9 +145,9 @@ public sealed class IndexController : ControllerBase
private Stream? StreamFromArchive(FileEntry fileEntry, string titleId, out IDisposable? streamContainer) private Stream? StreamFromArchive(FileEntry fileEntry, string titleId, out IDisposable? streamContainer)
{ {
// Example: file is inside an archive use ArchiveHandler // Example: file is inside an archive use ArchiveHandler
var archivePath = fileEntry.Path.Split(_snapshotService.GetArchivePathSeparator()).First(); var archivePath = fileEntry.Path.Split(snapshotService.GetArchivePathSeparator()).First();
_snapshotService.GetArchiveName(titleId); snapshotService.GetArchiveName(titleId);
var innerFileName = Path.GetFileName(fileEntry.Path.Split(_snapshotService.GetArchivePathSeparator()).Last()); var innerFileName = Path.GetFileName(fileEntry.Path.Split(snapshotService.GetArchivePathSeparator()).Last());
// Use SharpCompress to open the archive and find the entry. // Use SharpCompress to open the archive and find the entry.
// Only the 3 archive types we support are handled. // Only the 3 archive types we support are handled.
@@ -212,44 +194,4 @@ public sealed class IndexController : ControllerBase
streamContainer = null; streamContainer = null;
return null; return null;
} }
public class DependentStream : Stream
{
private readonly Stream _innerStream;
private readonly IDisposable? _parentContainer;
public DependentStream(Stream innerStream, IDisposable? parentContainer)
{
_innerStream = innerStream;
_parentContainer = parentContainer;
}
public override void Flush() => _innerStream.Flush();
public override int Read(byte[] buffer, int offset, int count) => _innerStream.Read(buffer, offset, count);
public override long Seek(long offset, SeekOrigin origin) => _innerStream.Seek(offset, origin);
public override void SetLength(long value) => _innerStream.SetLength(value);
public override void Write(byte[] buffer, int offset, int count) => _innerStream.Write(buffer, offset, count);
public override Task CopyToAsync(Stream destination, int bufferSize, CancellationToken cancellationToken)
{
return _innerStream.CopyToAsync(destination, bufferSize, cancellationToken);
}
public override bool CanRead => _innerStream.CanRead;
public override bool CanSeek => _innerStream.CanSeek;
public override bool CanWrite => _innerStream.CanWrite;
public override long Length => _innerStream.Length;
public override long Position { get => _innerStream.Position; set => _innerStream.Position = value; }
protected override void Dispose(bool disposing)
{
_parentContainer?.Dispose();
base.Dispose(disposing);
}
}
} }
@@ -4,24 +4,17 @@ using TinfoilVibeServer.Authentication;
namespace TinfoilVibeServer.Middleware; namespace TinfoilVibeServer.Middleware;
/// <summary> /// <summary>
/// Minimal BasicAuth middleware that also checks UID, failure counters and a blacklist. /// Minimal BasicAuth middleware that also checks UID, failure counters, and a blacklist.
/// </summary> /// </summary>
public sealed class BasicAuthMiddleware public sealed class BasicAuthMiddleware(RequestDelegate next)
{ {
private readonly RequestDelegate _next;
public BasicAuthMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task InvokeAsync(HttpContext context, IAuthStore store, ILogger<BasicAuthMiddleware> logger) public async Task InvokeAsync(HttpContext context, IAuthStore store, ILogger<BasicAuthMiddleware> logger)
{ {
// ------------- 1) Bypass auth for every path except “/” ---------------- // ------------- 1) Bypass auth for every path except “/” ----------------
// PathString is a struct compare its value directly. // PathString is a struct compare its value directly.
if (!context.Request.Path.Equals("/", StringComparison.Ordinal)) if (!context.Request.Path.Equals("/", StringComparison.Ordinal))
{ {
await _next(context); await next(context);
return; return;
} }
@@ -42,7 +35,7 @@ public sealed class BasicAuthMiddleware
{ {
logger.LogWarning("Missing Authorization header from {IP}", ip); logger.LogWarning("Missing Authorization header from {IP}", ip);
Challenge(context); Challenge(context);
logger.LogInformation("Sent 401 challenge to client"); logger.LogTrace("Sent 401 challenge to client");
return; return;
} }
@@ -50,14 +43,14 @@ public sealed class BasicAuthMiddleware
if (!authHeader.StartsWith("Basic ", StringComparison.OrdinalIgnoreCase)) if (!authHeader.StartsWith("Basic ", StringComparison.OrdinalIgnoreCase))
{ {
Challenge(context); Challenge(context);
logger.LogInformation("Sent 401 challenge to client"); logger.LogTrace("Sent 401 challenge to client");
return; return;
} }
string decoded; string decoded;
try try
{ {
var b64 = authHeader[6..].Trim(); var b64 = authHeader.Substring(6).Trim();
decoded = Encoding.UTF8.GetString(Convert.FromBase64String(b64)); decoded = Encoding.UTF8.GetString(Convert.FromBase64String(b64));
} }
catch catch
@@ -79,12 +72,9 @@ public sealed class BasicAuthMiddleware
var password = parts[1]; var password = parts[1];
// 3) UID header (optional) // 3) UID header (optional)
int? uid = null; string? uid = null;
if (context.Request.Headers.TryGetValue("UID", out var uidHeader)) if (context.Request.Headers.TryGetValue("UID", out var uidHeader))
{ uid = uidHeader.FirstOrDefault();
if (int.TryParse(uidHeader.ToString(), out var parsedUid))
uid = parsedUid;
}
// 4) Validate // 4) Validate
if (!store.TryValidate(username, password, uid, ip, out var error)) if (!store.TryValidate(username, password, uid, ip, out var error))
@@ -99,7 +89,7 @@ public sealed class BasicAuthMiddleware
// Authentication succeeded attach username for downstream handlers if needed // Authentication succeeded attach username for downstream handlers if needed
context.Items["User"] = username; context.Items["User"] = username;
logger.LogInformation("User {User} authenticated successfully (UID={UID})", username, uid); logger.LogInformation("User {User} authenticated successfully (UID={UID})", username, uid);
await _next(context); await next(context);
} }
private static void Challenge(HttpContext ctx) private static void Challenge(HttpContext ctx)
+1 -2
View File
@@ -10,6 +10,5 @@ public sealed record AppSettings(
string CredentialsFile, string CredentialsFile,
string FingerprintsFile, string FingerprintsFile,
string BlacklistFile, string BlacklistFile,
int MaxFailedAttempts, int MaxFailedAttempts
string KeySetFile
); );
+1 -1
View File
@@ -8,6 +8,6 @@ namespace TinfoilVibeServer.Models;
public sealed record FileEntry( public sealed record FileEntry(
string Path, // nsp or archive path string Path, // nsp or archive path
long Size, // size of nsp or full archive long Size, // size of nsp or full archive
string Hash, // SHA256 hex of first NCA of first NCP in NSP or archive string? Hash, // SHA256 hex of first NCA of first NCP in NSP or archive
List<NcaMetadataWithHash> Titles // Details of all NSP Roms in the Path List<NcaMetadataWithHash> Titles // Details of all NSP Roms in the Path
); );
@@ -2,5 +2,9 @@
public class IndexBuilderSettings public class IndexBuilderSettings
{ {
public string CacheFilePath { get; set; } = "indexcache.json";
public string ApiBaseUrl { get; set; } = "http://tinfoil.localhost"; public string ApiBaseUrl { get; set; } = "http://tinfoil.localhost";
public ICollection<string>? IndexDirectories { get; set; }
public string? LoginMessage { get; set; }
} }
@@ -0,0 +1,26 @@
using LibHac.Ncm;
namespace TinfoilVibeServer.Models;
/// <summary>
/// DTO returned by the extractor contains all data the snapshot needs.
/// </summary>
public sealed class NcaMetadataWithHash
{
public string TitleId { get; }
public string ApplicationTitle { get; set; }
public int Version { get; }
public ContentMetaType ContentMetaType { get; set; }
public string? Hash { get; }
public NcaMetadataWithHash(string titleId, string applicationTitle, int version,
ContentMetaType contentMetaType, string? hash = null)
{
TitleId = titleId;
ApplicationTitle = applicationTitle;
Version = version;
ContentMetaType = contentMetaType;
Hash = hash;
}
}
+16 -7
View File
@@ -1,36 +1,41 @@
using System.ComponentModel; using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
namespace TinfoilVibeServer.Models; namespace TinfoilVibeServer.Models;
public sealed class SnapshotOptions : INotifyPropertyChanged public sealed class SnapshotOptions : INotifyPropertyChanged
{ {
private List<string> _rootDirectories = new(); private List<string> _rootDirectories = [];
public List<string> RootDirectories public List<string> RootDirectories
{ {
get => _rootDirectories; get => _rootDirectories;
set set
{ {
if (_rootDirectories != value) if (_rootDirectories.Except(value) == Array.Empty<string>()) return;
{
_rootDirectories = value; _rootDirectories = value;
OnPropertyChanged(nameof(RootDirectories)); OnPropertyChanged(nameof(RootDirectories));
} }
} }
}
private List<string> _archiveExtensions = new(); private List<string> _archiveExtensions = new();
public List<string> ArchiveExtensions public List<string> ArchiveExtensions
{ {
get => _archiveExtensions; get => _archiveExtensions;
set set
{ {
if (_archiveExtensions != value) if (_archiveExtensions.Except(value) == Array.Empty<string>()) return;
{
_archiveExtensions = value; _archiveExtensions = value;
OnPropertyChanged(nameof(_archiveExtensions)); OnPropertyChanged(nameof(_archiveExtensions));
} }
} }
}
private List<string> _romExtensions = new(); private List<string> _romExtensions = new();
[Required(ErrorMessage = "No ROM extension specified")]
public List<string> RomExtensions public List<string> RomExtensions
{ {
get => _romExtensions; get => _romExtensions;
@@ -43,7 +48,9 @@ public sealed class SnapshotOptions : INotifyPropertyChanged
} }
} }
} }
private TimeSpan _cacheTtl = TimeSpan.FromHours(1); private TimeSpan _cacheTtl = TimeSpan.FromHours(1);
public TimeSpan CacheTtl public TimeSpan CacheTtl
{ {
get => _cacheTtl; get => _cacheTtl;
@@ -71,6 +78,7 @@ public sealed class SnapshotOptions : INotifyPropertyChanged
} }
private string _snapshotBackupFile = "snapshot.bak"; private string _snapshotBackupFile = "snapshot.bak";
public string SnapshotBackupFile public string SnapshotBackupFile
{ {
get => _snapshotBackupFile; get => _snapshotBackupFile;
@@ -83,6 +91,7 @@ public sealed class SnapshotOptions : INotifyPropertyChanged
} }
public event PropertyChangedEventHandler? PropertyChanged; public event PropertyChangedEventHandler? PropertyChanged;
private void OnPropertyChanged(string propertyName) => private void OnPropertyChanged(string propertyName) =>
PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(propertyName)); PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(propertyName));
} }
+44 -15
View File
@@ -1,3 +1,4 @@
using System.Text.Json;
using Microsoft.Extensions.Caching.Memory; using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Options; using Microsoft.Extensions.Options;
using TinfoilVibeServer.Authentication; using TinfoilVibeServer.Authentication;
@@ -9,31 +10,59 @@ var builder = WebApplication.CreateBuilder(args);
builder.Logging.ClearProviders(); builder.Logging.ClearProviders();
builder.Logging.AddConsole(); builder.Logging.AddConsole();
builder.Logging.AddDebug(); builder.Logging.AddDebug();
// -------------------------------------------------------------------
// 1) Configuration read appsettings.json once and expose it via
// ConfigManager (reloads on file change)
// -------------------------------------------------------------------
builder.Services.AddMemoryCache(); builder.Services.AddMemoryCache();
builder.Services.Configure<TitleDbOptions>(builder.Configuration.GetSection("TitleDb")); var dataRoot = builder.Configuration["CONFIG_ROOT"] ?? "/app/config/";
builder.Services.Configure<AuthSettings>(builder.Configuration.GetSection("AuthSettings")); // 1️⃣ Load the embedded default
builder.Services.Configure<SnapshotOptions>(builder.Configuration.GetSection("Snapshot")); var defaultResource = typeof(Program).Assembly
builder.Services.AddSingleton<ConfigManager>(); .GetManifestResourceStream("TinfoilVibeServer.appsettings.default.json")!; // adjust namespace
builder.Services.AddSingleton<INSPExtractor, NSPExtractor>(sp => var defaultConfig = JsonDocument.Parse(defaultResource).RootElement;
// 2️⃣ Try to write the file if it doesn't exist
var configPath = Path.Combine(dataRoot, "appsettings.json");
if (!File.Exists(configPath))
{ {
var config = sp.GetRequiredService<ConfigManager>(); // write the embedded JSON straight to disk
var logger = sp.GetRequiredService<ILogger<INSPExtractor>>(); try
var keySet = KeySetHolder.KeySet; // already loaded by ConfigManager {
return new NSPExtractor(keySet, logger); File.WriteAllText(configPath, defaultConfig.GetRawText());
}
catch (Exception e)
{
var tempFactory = LoggerFactory.Create(loggingBuilder =>
{
loggingBuilder.AddConsole();
loggingBuilder.AddDebug();
}); });
var logger = tempFactory.CreateLogger<Program>();
logger.LogError(e, "Failed to write default config file");
}
}
var config = new ConfigurationBuilder()
.AddJsonFile(Path.Combine(dataRoot,"appsettings.json"), optional: false, reloadOnChange: true)
.AddJsonFile(Path.Combine(dataRoot,$"appsettings.{builder.Environment.EnvironmentName}.json"), optional: true, reloadOnChange: true)
.AddJsonFile(Path.Combine(dataRoot,"appsettings.local.json"), optional: true, reloadOnChange: true)
.AddEnvironmentVariables()
.AddCommandLine(args).Build();
builder.Configuration.AddConfiguration(config);
builder.Services.AddOptions();
builder.Services.Configure<TitleDbOptions>(builder.Configuration.GetSection("TitleDb"));
builder.Services.Configure<NSPExtractorOptions>(builder.Configuration.GetSection("NSPExtractor"));
builder.Services.Configure<AuthSettings>(builder.Configuration.GetSection("AuthSettings"));
builder.Services.Configure<IndexBuilderSettings>(builder.Configuration.GetSection("IndexBuilder"));
builder.Services.AddOptions<SnapshotOptions>().Bind(builder.Configuration.GetSection("Snapshot")).ValidateOnStart();
builder.Services.AddSingleton<ConfigManager>();
builder.Services.AddSingleton<INSPExtractor, NSPExtractor>();
builder.Services.AddSingleton<SnapshotService>(); builder.Services.AddSingleton<SnapshotService>();
builder.Services.AddSingleton<ISnapshotService, SnapshotService>(sp => sp.GetRequiredService<SnapshotService>()); builder.Services.AddSingleton<ISnapshotService, SnapshotService>(sp => sp.GetRequiredService<SnapshotService>());
builder.Services.AddSingleton<IAuthStore, AuthStore>(); builder.Services.AddSingleton<IAuthStore, AuthStore>();
builder.Services.AddSingleton<TitleDatabaseService>(); builder.Services.AddSingleton<TitleDatabaseService>();
builder.Services.AddSingleton<IArchiveHandler, ArchiveHandler>(); builder.Services.AddSingleton<IArchiveHandler, ArchiveHandler>();
builder.Services.AddSingleton<IndexBuilderService>(); builder.Services.AddSingleton<IndexBuilderService>();
builder.Services.AddHostedService<IndexBuilderService>(provider => provider.GetRequiredService<IndexBuilderService>());
builder.Services.AddHostedService<SnapshotService>(provider => provider.GetRequiredService<SnapshotService>()); builder.Services.AddHostedService<SnapshotService>(provider => provider.GetRequiredService<SnapshotService>());
builder.Services.AddHostedService<TitleDatabaseService>(provider => provider.GetRequiredService<TitleDatabaseService>()).AddHttpClient(); builder.Services.AddHostedService<TitleDatabaseService>(provider => provider.GetRequiredService<TitleDatabaseService>()).AddHttpClient();
builder.Services.AddHostedService<IndexBuilderService>(provider => provider.GetRequiredService<IndexBuilderService>());
builder.Services.AddControllers(); // add MVC builder.Services.AddControllers(); // add MVC
// ------------------------------------------------------------------- // -------------------------------------------------------------------
// 2) Middleware BasicAuth (verifies username, password, UID, blacklist) // 2) Middleware BasicAuth (verifies username, password, UID, blacklist)
@@ -54,7 +83,7 @@ app.MapGet("/debug", () => new SnapshotService(
app.Services.GetRequiredService<IOptionsMonitor<SnapshotOptions>>(), app.Services.GetRequiredService<IOptionsMonitor<SnapshotOptions>>(),
app.Services.GetRequiredService<INSPExtractor>(), app.Services.GetRequiredService<INSPExtractor>(),
app.Services.GetRequiredService<IArchiveHandler>(), app.Services.GetRequiredService<IArchiveHandler>(),
app.Services.GetRequiredService<ILogger<SnapshotService>>()) app.Services.GetRequiredService<ILogger<SnapshotService>>(), app.Services.GetRequiredService<IHostEnvironment>())
.GetSnapshot()); .GetSnapshot());
app.Lifetime.ApplicationStarted.Register(() => app.Lifetime.ApplicationStarted.Register(() =>
app.Services.GetRequiredService<ILogger<Program>>().LogInformation("Application started. Listening on {Urls}", string.Join(", ", app.Urls))); app.Services.GetRequiredService<ILogger<Program>>().LogInformation("Application started. Listening on {Urls}", string.Join(", ", app.Urls)));
@@ -5,9 +5,10 @@
"commandName": "Project", "commandName": "Project",
"dotnetRunMessages": true, "dotnetRunMessages": true,
"launchBrowser": false, "launchBrowser": false,
"applicationUrl": "http://192.168.1.145:80;http://tinfoil.localhost:8080;http://tinfoil.ecenshu.net", "applicationUrl": "http://192.168.1.145:8080;http://tinfoil.localhost:8081;http://tinfoil.ecenshu.net",
"environmentVariables": { "environmentVariables": {
"ASPNETCORE_ENVIRONMENT": "Development" "ASPNETCORE_ENVIRONMENT": "Development",
"CONFIG_ROOT": "./config/"
} }
}, },
"https": { "https": {
+1 -1
View File
@@ -97,7 +97,7 @@ public sealed class ArchiveHandler : IArchiveHandler
using var archive = SevenZipArchive.Open(path); using var archive = SevenZipArchive.Open(path);
foreach (var entry in archive.Entries) foreach (var entry in archive.Entries)
{ {
if (!entry.IsDirectory && IsRomArchive(entry.Key)) if (!entry.IsDirectory && entry.Key != null && IsRomArchive(entry.Key))
{ {
var temp = Path.GetTempFileName(); var temp = Path.GetTempFileName();
entry.WriteToFile(temp); entry.WriteToFile(temp);
+9 -33
View File
@@ -1,5 +1,4 @@
using System.IO; using System.Text.Json;
using System.Text.Json;
using LibHac.Common.Keys; using LibHac.Common.Keys;
using TinfoilVibeServer.Models; using TinfoilVibeServer.Models;
@@ -11,17 +10,16 @@ namespace TinfoilVibeServer.Services;
/// </summary> /// </summary>
public class ConfigManager public class ConfigManager
{ {
public AppSettings Settings { get; private set; } public AppSettings? Settings { get; private set; }
public event Action<AppSettings?>? OnChange;
public event Action<AppSettings>? OnChange;
private readonly string _configPath; private readonly string _configPath;
private readonly FileSystemWatcher _watcher; private readonly FileSystemWatcher _watcher;
private readonly object _sync = new(); private readonly Lock _sync = new();
public ConfigManager() public ConfigManager()
{ {
_configPath = Path.Combine(AppContext.BaseDirectory, "appsettings.json"); _configPath = Path.Combine(AppContext.BaseDirectory, "config", "appsettings.json");
Load(); Load();
_watcher = new FileSystemWatcher _watcher = new FileSystemWatcher
@@ -39,41 +37,19 @@ public class ConfigManager
if (!File.Exists(_configPath)) if (!File.Exists(_configPath))
{ {
Settings = new AppSettings( Settings = new AppSettings(
RootDirectories: Array.Empty<string>(), RootDirectories: [],
WhitelistExtensions: Array.Empty<string>(), WhitelistExtensions: [],
RomExtensions: Array.Empty<string>(), RomExtensions: [],
CredentialsFile: "credentials.json", CredentialsFile: "credentials.json",
FingerprintsFile: "fingerprints.json", FingerprintsFile: "fingerprints.json",
BlacklistFile: "blacklist.json", BlacklistFile: "blacklist.json",
MaxFailedAttempts: 5, MaxFailedAttempts: 5
KeySetFile: "keys.bin"
); );
return; return;
} }
var txt = File.ReadAllText(_configPath); var txt = File.ReadAllText(_configPath);
Settings = JsonSerializer.Deserialize<AppSettings>(txt, new JsonSerializerOptions { PropertyNameCaseInsensitive = true })!; Settings = JsonSerializer.Deserialize<AppSettings>(txt, new JsonSerializerOptions { PropertyNameCaseInsensitive = true })!;
// --- Load the KeySet --------------------------------------------
if (!string.IsNullOrWhiteSpace(Settings.KeySetFile))
{
var keyFilePath = Path.Combine(AppContext.BaseDirectory, Settings.KeySetFile);
if (File.Exists(keyFilePath))
{
// LibHac provides a static helper to load a keyset file.
// If the file is not found or corrupt, we simply keep the
// default (empty) key set the app will throw later
// when a title requires a missing key.
try
{
KeySetHolder.KeySet = ExternalKeyReader.ReadKeyFile(keyFilePath);
}
catch
{
KeySetHolder.KeySet = new KeySet(); // fallback
}
}
}
} }
private void Reload() private void Reload()
@@ -11,29 +11,35 @@ namespace TinfoilVibeServer.Services;
// *** NEW *** // *** NEW ***
public sealed class IndexBuilderService: IHostedService public sealed class IndexBuilderService: IHostedService
{ {
private const string CacheFileName = "indexcache.json"; private readonly IOptionsMonitor<IndexBuilderSettings> _options;
private readonly IOptions<IndexBuilderSettings> _options;
private readonly ISnapshotService _snapshotService; private readonly ISnapshotService _snapshotService;
private readonly TitleDatabaseService _titleDb; private readonly TitleDatabaseService _titleDb;
private readonly IConfiguration _configuration;
private readonly ILogger<IndexBuilderService> _logger; private readonly ILogger<IndexBuilderService> _logger;
private readonly string _cachePath; private readonly IHostEnvironment _hostEnvironment;
private readonly SemaphoreSlim _lock = new(1, 1); private readonly SemaphoreSlim _lock = new(1, 1);
public IndexBuilderService( public IndexBuilderService(
IOptions<IndexBuilderSettings> options, IOptionsMonitor<IndexBuilderSettings> options,
ISnapshotService snapshotService, ISnapshotService snapshotService,
TitleDatabaseService titleDb, TitleDatabaseService titleDb,
IConfiguration configuration, ILogger<IndexBuilderService> logger,
ILogger<IndexBuilderService> logger) IHostEnvironment hostEnvironment)
{ {
_options = options; _options = options;
_snapshotService = snapshotService; _snapshotService = snapshotService;
_titleDb = titleDb; _titleDb = titleDb;
_configuration = configuration;
_logger = logger; _logger = logger;
_cachePath = Path.Combine(AppContext.BaseDirectory, CacheFileName); _hostEnvironment = hostEnvironment;
}
private static string DetermineCachePath(IOptionsMonitor<IndexBuilderSettings> options, IHostEnvironment environment)
{
if (Path.IsPathRooted(options.CurrentValue.CacheFilePath))
{
return options.CurrentValue.CacheFilePath;
}
return Path.Combine(environment.ContentRootPath, "data", options.CurrentValue.CacheFilePath);
} }
public IndexDto Build(HttpContext httpContext) public IndexDto Build(HttpContext httpContext)
@@ -60,10 +66,9 @@ public sealed class IndexBuilderService: IHostedService
// 3️⃣ Build new index from snapshot entries // 3️⃣ Build new index from snapshot entries
var files = ParseSnapshotFiles(snapshot, new Uri(httpContext.Request.Scheme + "://" + httpContext.Request.Host + httpContext.Request.PathBase)); var files = ParseSnapshotFiles(snapshot, new Uri(httpContext.Request.Scheme + "://" + httpContext.Request.Host + httpContext.Request.PathBase));
var directories = _configuration.GetSection("Directories") var directories = _options.CurrentValue.IndexDirectories ?? Array.Empty<string>();
.Get<string[]>() ?? Array.Empty<string>();
var success = _configuration["SuccessMessage"] ?? string.Empty; var success = _options.CurrentValue.LoginMessage ?? string.Empty;
var index = new IndexDto(files.SelectMany(inner => inner).ToList(), directories.ToList(), success); var index = new IndexDto(files.SelectMany(inner => inner).ToList(), directories.ToList(), success);
@@ -114,9 +119,11 @@ public sealed class IndexBuilderService: IHostedService
var fileName = Uri.EscapeDataString($"{name}[{titleId}][v{versionNumberParsed}][{patchOrApp}].nsp"); var fileName = Uri.EscapeDataString($"{name}[{titleId}][v{versionNumberParsed}][{patchOrApp}].nsp");
var url = $"{baseUri.ToString().TrimEnd('/')}/{fileName}"; var url = $"{baseUri.ToString().TrimEnd('/')}/{fileName}";
if (Uri.IsWellFormedUriString(url, UriKind.Absolute)) var isWellFormed = Uri.TryCreate(url, UriKind.Absolute, out var parsedUri);
if (isWellFormed && parsedUri != null)
{ {
fileDtos.Add(new FileDto(url, e.Size)); fileDtos.Add(new FileDto(parsedUri.AbsoluteUri, e.Size));
} }
else else
{ {
@@ -133,18 +140,28 @@ public sealed class IndexBuilderService: IHostedService
private IndexCache? LoadCache() private IndexCache? LoadCache()
{ {
if (!File.Exists(_cachePath)) return null; var cachePath = DetermineCachePath(_options, _hostEnvironment);
if (!File.Exists(cachePath)) return null;
_lock.Wait(); _lock.Wait();
var json = File.ReadAllText(_cachePath); var json = File.ReadAllText(cachePath);
_lock.Release(); _lock.Release();
_logger.LogInformation("Loaded index cache from {Path}", _cachePath); _logger.LogInformation("Loaded index cache from {Path}", cachePath);
return JsonSerializer.Deserialize<IndexCache>(json); return JsonSerializer.Deserialize<IndexCache>(json);
} }
private void PersistCache(string snapshotHash, IndexDto index) private void PersistCache(string snapshotHash, IndexDto index)
{ {
var cachePath = DetermineCachePath(_options, _hostEnvironment);
var cache = new IndexCache(snapshotHash, index); var cache = new IndexCache(snapshotHash, index);
File.WriteAllText(_cachePath, JsonSerializer.Serialize(cache, new JsonSerializerOptions{WriteIndented=true})); File.WriteAllText(cachePath, JsonSerializer.Serialize(cache, new JsonSerializerOptions{WriteIndented=true}));
}
public void InvalidateIndex(object? sender, EventArgs e)
{
var cachePath = DetermineCachePath(_options, _hostEnvironment);
if (!File.Exists(cachePath)) return;
File.Delete(cachePath);
_logger.LogInformation("Index cache cleared");
} }
private static string ComputeSnapshotHash(IEnumerable<FileEntry> entries) private static string ComputeSnapshotHash(IEnumerable<FileEntry> entries)
@@ -161,27 +178,17 @@ public sealed class IndexBuilderService: IHostedService
public Task StartAsync(CancellationToken cancellationToken) public Task StartAsync(CancellationToken cancellationToken)
{ {
var url = new Uri(_options.Value.ApiBaseUrl); var url = new Uri(_options.CurrentValue.ApiBaseUrl);
var host = url.Host; var host = url.Host;
Build(new DefaultHttpContext {HttpContext = { Request = { Host = new HostString(host), Scheme = url.Scheme, Path = new PathString(url.AbsolutePath)}}}); Build(new DefaultHttpContext {HttpContext = { Request = { Host = new HostString(host), Scheme = url.Scheme, Path = new PathString(url.AbsolutePath)}}});
_snapshotService.SnapshotRebuilt += InvalidateIndex; _snapshotService.SnapshotRebuilt += InvalidateIndex;
return Task.CompletedTask; return Task.CompletedTask;
} }
public void InvalidateIndex(object? sender, EventArgs e)
{
if (!File.Exists(_cachePath)) return;
File.Delete(_cachePath);
_logger.LogInformation("Index cache cleared");
}
public Task StopAsync(CancellationToken cancellationToken) public Task StopAsync(CancellationToken cancellationToken)
{ {
_snapshotService.SnapshotRebuilt -= InvalidateIndex; _snapshotService.SnapshotRebuilt -= InvalidateIndex;
return Task.CompletedTask; // nothing special to do on shutdown return Task.CompletedTask; // nothing special to do on shutdown
} }
#endregion #endregion
} }
+58 -29
View File
@@ -9,6 +9,9 @@ using LibHac.Common.Keys;
using LibHac.Ncm; using LibHac.Ncm;
using LibHac.Tools.Fs; using LibHac.Tools.Fs;
using LibHac.Tools.Ncm; using LibHac.Tools.Ncm;
using Microsoft.Extensions.Options;
using TinfoilVibeServer.Models;
using Path = System.IO.Path;
namespace TinfoilVibeServer.Services namespace TinfoilVibeServer.Services
{ {
@@ -32,13 +35,49 @@ namespace TinfoilVibeServer.Services
/// </summary> /// </summary>
public sealed class NSPExtractor : INSPExtractor public sealed class NSPExtractor : INSPExtractor
{ {
private readonly KeySet _keySet; private KeySet? _keySet;
private readonly ILogger<INSPExtractor> _logger;
public NSPExtractor(KeySet keySet, ILogger<INSPExtractor> logger) public KeySet? KeySet
{ {
_keySet = keySet; get
{
if (_keySet != null) return _keySet;
if (_options.CurrentValue.KeyFile == null) return null;
var dataRoot = _environment.ContentRootPath ?? "/app/config";
if (Path.IsPathRooted(_options.CurrentValue.KeyFile))
{
_keySet = ExternalKeyReader.ReadKeyFile(_options.CurrentValue.KeyFile);
}
else
{
_keySet = ExternalKeyReader.ReadKeyFile(Path.Combine(dataRoot, "config", _options.CurrentValue.KeyFile));
}
return _keySet;
}
}
private readonly IOptionsMonitor<NSPExtractorOptions> _options;
private readonly ILogger<INSPExtractor> _logger;
private readonly IHostEnvironment _environment;
public NSPExtractor(IOptionsMonitor<NSPExtractorOptions> options, ILogger<INSPExtractor> logger, IHostEnvironment environment)
{
_options = options;
_options.OnChange(o =>
{
if (o.KeyFile == null)
{
_logger?.LogInformation("No KeySet specified, skipping key validation");
}
if (!File.Exists(o.KeyFile))
{
_logger?.LogWarning("KeySet file {KeyFile} does not exist", o.KeyFile);
}
});
_logger = logger; _logger = logger;
_environment = environment;
} }
/// <summary> /// <summary>
@@ -55,6 +94,8 @@ namespace TinfoilVibeServer.Services
/// </summary> /// </summary>
public NcaMetadataWithHash? ExtractFromStream(Stream stream) public NcaMetadataWithHash? ExtractFromStream(Stream stream)
{ {
if (KeySet == null) return null;
if (!stream.CanSeek) return null; if (!stream.CanSeek) return null;
stream.Seek(0, SeekOrigin.Begin); stream.Seek(0, SeekOrigin.Begin);
@@ -67,7 +108,7 @@ namespace TinfoilVibeServer.Services
if (IsXciFileSystem(stream)) if (IsXciFileSystem(stream))
{ {
var xci = new Xci(_keySet, storage); var xci = new Xci(KeySet, storage);
List<DirectoryEntryEx> ncaEntries; List<DirectoryEntryEx> ncaEntries;
if (xci.HasPartition(XciPartitionType.Secure)) if (xci.HasPartition(XciPartitionType.Secure))
{ {
@@ -85,7 +126,7 @@ namespace TinfoilVibeServer.Services
using var ncaFile = fileRef.Release(); using var ncaFile = fileRef.Release();
using var ncaFileStorage = new FileStorage(ncaFile); using var ncaFileStorage = new FileStorage(ncaFile);
var nca = new Nca(_keySet, ncaFileStorage); var nca = new Nca(KeySet, ncaFileStorage);
if (hash == null) if (hash == null)
{ {
// Hash the *first* NCA stream the stream we just opened // Hash the *first* NCA stream the stream we just opened
@@ -112,6 +153,8 @@ namespace TinfoilVibeServer.Services
private NcaMetadataWithHash? ExtractNSPFromStream(StreamStorage storage) private NcaMetadataWithHash? ExtractNSPFromStream(StreamStorage storage)
{ {
if (KeySet == null) return null;
List<DirectoryEntryEx> ncaEntries; List<DirectoryEntryEx> ncaEntries;
_logger.LogInformation("Processing as NSP"); _logger.LogInformation("Processing as NSP");
var partition = new PartitionFileSystem(); var partition = new PartitionFileSystem();
@@ -129,7 +172,7 @@ namespace TinfoilVibeServer.Services
using var ncaFile = fileRef.Release(); using var ncaFile = fileRef.Release();
using var ncaFileStorage = new FileStorage(ncaFile); using var ncaFileStorage = new FileStorage(ncaFile);
var nca = new Nca(_keySet, ncaFileStorage); var nca = new Nca(KeySet, ncaFileStorage);
if (hash == null) if (hash == null)
{ {
// Hash the *first* NCA stream the stream we just opened // Hash the *first* NCA stream the stream we just opened
@@ -198,6 +241,8 @@ namespace TinfoilVibeServer.Services
} }
private bool IsXciFileSystem(Stream stream) private bool IsXciFileSystem(Stream stream)
{ {
if (KeySet == null) return false;
try try
{ {
if (!stream.CanSeek) return false; if (!stream.CanSeek) return false;
@@ -206,7 +251,7 @@ namespace TinfoilVibeServer.Services
var storage = new StreamStorage(stream, true); var storage = new StreamStorage(stream, true);
try try
{ {
var xciBlock = new Xci(_keySet, storage); var xciBlock = new Xci(KeySet, storage);
_logger.LogInformation("XCI found"); _logger.LogInformation("XCI found");
return xciBlock.HasPartition(XciPartitionType.Secure); return xciBlock.HasPartition(XciPartitionType.Secure);
} }
@@ -225,6 +270,8 @@ namespace TinfoilVibeServer.Services
public string ExtractHashFromStream(Stream nspStream) public string ExtractHashFromStream(Stream nspStream)
{ {
if (KeySet == null) return string.Empty;
if (!IsPfs0FileSystem(nspStream)) if (!IsPfs0FileSystem(nspStream))
return string.Empty; return string.Empty;
@@ -249,7 +296,7 @@ namespace TinfoilVibeServer.Services
try try
{ {
var nca = new Nca(_keySet, ncaFileStorage); var nca = new Nca(KeySet, ncaFileStorage);
if (nca.Header.ContentType != NcaContentType.Meta) if (nca.Header.ContentType != NcaContentType.Meta)
continue; // only the meta NCA contains title metadata continue; // only the meta NCA contains title metadata
@@ -271,26 +318,8 @@ namespace TinfoilVibeServer.Services
} }
} }
/// <summary> public class NSPExtractorOptions
/// DTO returned by the extractor contains all data the snapshot needs.
/// </summary>
public sealed class NcaMetadataWithHash
{ {
public string TitleId { get; } public string? KeyFile { get; set; }
public string ApplicationTitle { get; set; }
public int Version { get; }
public ContentMetaType ContentMetaType { get; set; }
public string Hash { get; }
public NcaMetadataWithHash(string titleId, string applicationTitle, int version,
ContentMetaType contentMetaType, string hash)
{
TitleId = titleId;
ApplicationTitle = applicationTitle;
Version = version;
ContentMetaType = contentMetaType;
Hash = hash;
}
} }
} }
@@ -83,7 +83,7 @@ namespace TinfoilVibeServer.Services
continue; continue;
// SharpCompress gives us a stream that must be disposed by the caller // SharpCompress gives us a stream that must be disposed by the caller
yield return new RomArchiveEntry(entry.Key, entry.OpenEntryStream()); if (entry.Key != null) yield return new RomArchiveEntry(entry.Key, entry.OpenEntryStream());
} }
} }
} }
+241 -104
View File
@@ -3,13 +3,14 @@ using System.Security.Cryptography;
using System.Text.Json; using System.Text.Json;
using Microsoft.Extensions.Caching.Memory; using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Options; using Microsoft.Extensions.Options;
using Microsoft.Extensions.Primitives;
using TinfoilVibeServer.Models; using TinfoilVibeServer.Models;
using TinfoilVibeServer.Utilities; using TinfoilVibeServer.Utilities;
namespace TinfoilVibeServer.Services; namespace TinfoilVibeServer.Services;
public interface ISnapshotService public interface ISnapshotService
{ {
event EventHandler SnapshotRebuilt; // raised after a rebuild event EventHandler SnapshotRebuilt; // event raised after a rebuild
void RebuildSnapshot(); void RebuildSnapshot();
SnapshotService.ROMSnapshot GetSnapshot(); SnapshotService.ROMSnapshot GetSnapshot();
@@ -26,19 +27,32 @@ public interface ISnapshotService
public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedService public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedService
{ {
#region FileSystemWatcher #region FileSystemWatcher
/* ==============================================================
* 1️⃣ FileSystemWatcher
* ============================================================== */
private readonly List<FileSystemWatcher> _watchers = new(); private readonly List<FileSystemWatcher> _watchers = new();
#endregion #endregion
#region Snapshot options & helpers
/* ==============================================================
* 2️⃣ Snapshot options & helpers
* ============================================================== */
private readonly SnapshotOptions _options; private readonly SnapshotOptions _options;
private readonly INSPExtractor _nspExtractor; private readonly INSPExtractor _nspExtractor;
private readonly IArchiveHandler _archiveHandler; private readonly IArchiveHandler _archiveHandler;
private readonly ILogger<SnapshotService> _logger; private readonly ILogger<SnapshotService> _logger;
private readonly IHostEnvironment _environment;
private readonly string _jsonPath; private readonly string _jsonPath;
private readonly string _snapshotPath; private readonly string _snapshotPath;
private readonly ConcurrentDictionary<string, SnapshotEntry> _cache = new(); private readonly ConcurrentDictionary<string, SnapshotEntry> _cache = new();
private readonly ConcurrentDictionary<string, string> _hashCache = new(); private readonly ConcurrentDictionary<string, string> _hashCache = new();
// Archive full path -> FileEntry.Path // Archive full path -> FileEntry.Path
private readonly ConcurrentDictionary<string, string> _archiveLookup = new(); private readonly ConcurrentDictionary<string, string> _archiveLookup = new();
// hash -> file size // hash -> file size
private readonly ConcurrentDictionary<string, long> _sizeLookup = new(); private readonly ConcurrentDictionary<string, long> _sizeLookup = new();
private readonly IMemoryCache _debouncerCache; private readonly IMemoryCache _debouncerCache;
@@ -47,59 +61,75 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
private readonly SemaphoreSlim _snapshotFileSemaphore = new(1, 1); private readonly SemaphoreSlim _snapshotFileSemaphore = new(1, 1);
private const char ArchivePathSeparator = '|'; private const char ArchivePathSeparator = '|';
public char GetArchivePathSeparator() => ArchivePathSeparator;
public char GetArchivePathSeparator() => ArchivePathSeparator;
#endregion
/* ==============================================================
* 3️⃣ Buildtime guard
* ============================================================== */
/// <summary>
/// Allows only one rebuild at a time.
/// </summary>
private readonly SemaphoreSlim _buildLock = new(1, 1);
/* ==============================================================
* 4️⃣ Constructor
* ============================================================== */
public SnapshotService( public SnapshotService(
IMemoryCache debouncerCache, IMemoryCache debouncerCache,
IOptionsMonitor<SnapshotOptions> options, IOptionsMonitor<SnapshotOptions> options,
INSPExtractor nspExtractor, INSPExtractor nspExtractor,
IArchiveHandler archiveHandler, IArchiveHandler archiveHandler,
ILogger<SnapshotService> logger) ILogger<SnapshotService> logger,
IHostEnvironment environment
)
{ {
_options = options.CurrentValue; _options = options.CurrentValue;
_debouncerCache = debouncerCache; _debouncerCache = debouncerCache;
_nspExtractor = nspExtractor; _nspExtractor = nspExtractor;
_archiveHandler = archiveHandler; _archiveHandler = archiveHandler;
_logger = logger; _logger = logger;
_jsonPath = Path.Combine(AppContext.BaseDirectory, _options.SnapshotFile); _environment = environment;
_jsonPath = Path.Combine(Path.DirectorySeparatorChar.ToString(), "app", "data", _options.SnapshotFile);
// Debounce timer for persisting snapshot FileSystemExtensions.EnsureDirectoryExists(Path.GetFullPath(Path.GetDirectoryName(_jsonPath) ?? throw new InvalidOperationException()));
long debounceTime = 200;
var entryOptions = new MemoryCacheEntryOptions()
.SetSlidingExpiration(TimeSpan.FromSeconds(debounceTime)).RegisterPostEvictionCallback((key, value, reason,
state) =>
{
_logger.LogInformation("Should persist the snapshot {Key}, {Reason}", key, reason);
}); // <‑‑ sliding!
FileSystemExtensions.EnsureDirectoryExists(Path.GetFullPath(Path.GetDirectoryName(_jsonPath)));
if (!File.Exists(_jsonPath)) if (!File.Exists(_jsonPath))
{ {
_snapshotFileSemaphore.Wait(); _snapshotFileSemaphore.Wait();
File.WriteAllText(_jsonPath, "[]"); File.WriteAllText(_jsonPath, "[]");
_snapshotFileSemaphore.Release(); _snapshotFileSemaphore.Release();
} }
_snapshotPath = Path.Combine(AppContext.BaseDirectory, _options.SnapshotBackupFile);
FileSystemExtensions.EnsureDirectoryExists(Path.GetFullPath(Path.GetDirectoryName(_snapshotPath))); _snapshotPath = Path.Combine(Path.DirectorySeparatorChar.ToString(), "app", "data", _options.SnapshotBackupFile);
FileSystemExtensions.EnsureDirectoryExists(Path.GetFullPath(Path.GetDirectoryName(_snapshotPath) ?? throw new InvalidOperationException()));
// 1️⃣ Register for *property* changes // 1️⃣ Register for *property* changes
_options.PropertyChanged += (s, e) => OnOptionsChanged(e.PropertyName); options.OnChange((snapshotOptions, _) => { _options.RootDirectories = snapshotOptions.RootDirectories; });
_options.PropertyChanged += (_, e) => OnOptionsChanged(e.PropertyName);
if (_options.RootDirectories.Count == 0)
{
_logger.LogInformation("No directories set to watch for ROMS/Archives");
}
foreach (var path in _options.RootDirectories) foreach (var path in _options.RootDirectories)
{ {
AddWatchDirectory(path); AddWatchDirectory(path);
} }
} }
// --------- Private helpers --------- // --------- Private helpers ---------
private void OnOptionsChanged(string propertyName) private void OnOptionsChanged(string? propertyName)
{
if (propertyName == nameof(SnapshotOptions.RootDirectories))
{ {
if (propertyName != nameof(SnapshotOptions.RootDirectories)) return;
_logger.LogInformation("Root directories changed, rebuilding snapshot");
var fileSystemWatchers = _watchers.Where(watcher => !_options.RootDirectories.Contains(watcher.Path)); var fileSystemWatchers = _watchers.Where(watcher => !_options.RootDirectories.Contains(watcher.Path));
foreach (var watcher in fileSystemWatchers) var systemWatchers = fileSystemWatchers.ToList();
foreach (var watcher in systemWatchers)
{ {
watcher.EnableRaisingEvents = false; RemoveWatchDirectory(watcher.Path);
watcher.Dispose();
_watchers.Remove(watcher);
} }
var newWatchedDirectories = _options.RootDirectories.Where(newWatchedDirectory => var newWatchedDirectories = _options.RootDirectories.Where(newWatchedDirectory =>
@@ -109,16 +139,18 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
foreach (var newWatchedDirectory in newWatchedDirectories) foreach (var newWatchedDirectory in newWatchedDirectories)
{ {
AddWatchDirectory(newWatchedDirectory); AddWatchDirectory(newWatchedDirectory);
} }
BuildSnapshotAsync(); // rebuild everything BuildSnapshotAsync(); // rebuild everything
PersistSnapshotAsync(); PersistSnapshotAsync();
} }
}
#region FileSystemWatcher #region FileSystemWatcher
/* ==============================================================
* 5️⃣ FileSystemWatcher helpers
* ============================================================== */
private void AddWatchDirectory(string path) private void AddWatchDirectory(string path)
{ {
if (!Directory.Exists(path)) return; if (!Directory.Exists(path)) return;
@@ -153,9 +185,20 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
private void ThrottleSnapshotUpdate(FileSystemEventArgs fileSystemEventArgs) private void ThrottleSnapshotUpdate(FileSystemEventArgs fileSystemEventArgs)
{ {
// If a rebuild is in progress, ignore the event immediately
if (_buildLock.CurrentCount == 0) // lock held by a rebuild
{
_logger.LogInformation(
"File system event {ChangeType} on {Path} ignored because a rebuild is already in progress",
fileSystemEventArgs.ChangeType, fileSystemEventArgs.FullPath);
return;
}
SnapshotRebuilding?.Invoke(this, fileSystemEventArgs); SnapshotRebuilding?.Invoke(this, fileSystemEventArgs);
CancellationTokenSource cts = new();
using var cacheEntry = _debouncerCache.CreateEntry(fileSystemEventArgs.FullPath) using var cacheEntry = _debouncerCache.CreateEntry(fileSystemEventArgs.FullPath)
//.SetAbsoluteExpiration(TimeSpan.FromMilliseconds(DebounceMs)) .AddExpirationToken(new CancellationChangeToken(cts.Token))
.SetValue(fileSystemEventArgs) .SetValue(fileSystemEventArgs)
.SetOptions(new MemoryCacheEntryOptions .SetOptions(new MemoryCacheEntryOptions
{ {
@@ -164,9 +207,9 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
new PostEvictionCallbackRegistration new PostEvictionCallbackRegistration
{ {
EvictionCallback = EvictionCallback =
(key, value, reason, state) => (_, value, reason, _) =>
{ {
if (reason != EvictionReason.Expired) return; if (reason is not (EvictionReason.Expired or EvictionReason.TokenExpired)) return;
if (value is FileSystemEventArgs args) if (value is FileSystemEventArgs args)
{ {
@@ -174,8 +217,9 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
{ {
_logger.LogInformation("File {FilePath} is locked, skipping snapshot update", args.FullPath); _logger.LogInformation("File {FilePath} is locked, skipping snapshot update", args.FullPath);
using var rebounce = _debouncerCache.CreateEntry(args.FullPath) using var rebounce = _debouncerCache.CreateEntry(args.FullPath)
.SetAbsoluteExpiration(TimeSpan.FromMilliseconds(DebounceMs)) .AddExpirationToken(new CancellationChangeToken(cts.Token))
.SetValue(args); .SetValue(args);
cts.CancelAfter(TimeSpan.FromMilliseconds(DebounceMs));
} }
} }
@@ -184,11 +228,11 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
} }
} }
}); });
cacheEntry.AbsoluteExpirationRelativeToNow = TimeSpan.FromMilliseconds(DebounceMs); cts.CancelAfter(TimeSpan.FromMilliseconds(DebounceMs));
_logger.LogDebug("File system event {EventType} on {Path} at {Time}", fileSystemEventArgs.ChangeType, _logger.LogDebug("File system event {EventType} on {Path} at {Time}", fileSystemEventArgs.ChangeType,
fileSystemEventArgs.FullPath, DateTime.Now.ToString("HH:mm:ss")); fileSystemEventArgs.FullPath, DateTime.Now.ToString("HH:mm:ss.fff"));
} }
private static bool IsFileLocked(string filePath) private static bool IsFileLocked(string filePath)
{ {
FileStream? stream = null; FileStream? stream = null;
@@ -206,6 +250,7 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
{ {
stream?.Close(); stream?.Close();
} }
return false; return false;
} }
@@ -225,9 +270,19 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
public Task AddToSnapshotAsync(FileEntry entry) public Task AddToSnapshotAsync(FileEntry entry)
{ {
// Update lookup tables // Update lookup tables
_cache[entry.Path] = new SnapshotEntry(entry.Path, entry.Hash, entry.Size, entry.Titles); if (entry.Hash != null)
{
var lastModified = File.GetLastWriteTimeUtc(entry.Path);
_cache[entry.Path] = new SnapshotEntry(entry.Path, entry.Hash, entry.Size, lastModified, entry.Titles);
_hashCache[entry.Hash] = entry.Path; _hashCache[entry.Hash] = entry.Path;
_sizeLookup[entry.Hash] = entry.Size; _sizeLookup[entry.Hash] = entry.Size;
}
else
{
_logger.LogWarning("Cannot add entry {Path} to snapshot: no hash", entry.Path);
return Task.CompletedTask;
}
if (entry.Path.Contains(ArchivePathSeparator)) if (entry.Path.Contains(ArchivePathSeparator))
{ {
var filename = entry.Path.Split(ArchivePathSeparator)[0]; var filename = entry.Path.Split(ArchivePathSeparator)[0];
@@ -236,93 +291,102 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
foreach (var ncaMetadataWithHash in entry.Titles) foreach (var ncaMetadataWithHash in entry.Titles)
{ {
if (ncaMetadataWithHash.Hash == null)
{
_logger.LogWarning("Cannot add entry {Path} to snapshot: no hash", entry.Path);
continue;
}
_hashCache[ncaMetadataWithHash.Hash] = entry.Path; _hashCache[ncaMetadataWithHash.Hash] = entry.Path;
_sizeLookup[ncaMetadataWithHash.Hash] = entry.Size; _sizeLookup[ncaMetadataWithHash.Hash] = entry.Size;
_logger.LogInformation("Added entry {titleId} to snapshot (hash={hash})", ncaMetadataWithHash.TitleId, ncaMetadataWithHash.Hash); _logger.LogInformation("Added entry {titleId} to snapshot (hash={hash})", ncaMetadataWithHash.TitleId, ncaMetadataWithHash.Hash);
} }
// Persist snapshot to disk // Persist snapshot to disk
PersistSnapshotAsync(); PersistSnapshotAsync();
return Task.CompletedTask; return Task.CompletedTask;
} }
/* ==============================================================
* 6️⃣ Snapshot build / persistence helpers
* ============================================================== */
/// Builds _cache and _hashCache based on directory configuration /// Builds _cache and _hashCache based on directory configuration
public Task BuildSnapshotAsync() public Task BuildSnapshotAsync()
{
// Acquire the rebuild lock if we cannot, skip this build.
if (!_buildLock.Wait(0))
{
_logger.LogInformation("BuildSnapshotAsync called while rebuild in progress, ignoring.");
return Task.CompletedTask;
}
try
{ {
_logger.LogInformation("Building snapshot"); _logger.LogInformation("Building snapshot");
var index = LoadSnapshotIndex(); var index = LoadSnapshotIndex();
var latestModifiedUtcParallel = FileSystemExtensions.GetLatestModifiedUtcParallel(_options.RootDirectories); var latestModifiedUtcParallel = FileSystemExtensions.GetLatestModifiedUtcParallel(_options.RootDirectories);
var fileInfo = new FileInfo(_snapshotPath); var fileInfo = new FileInfo(_snapshotPath);
bool snapshotVerified = true; bool snapshotVerified = fileInfo.Exists;
if (latestModifiedUtcParallel.HasValue && latestModifiedUtcParallel.Value < fileInfo.LastWriteTimeUtc) if (latestModifiedUtcParallel.HasValue && latestModifiedUtcParallel.Value < fileInfo.LastWriteTimeUtc)
{ {
if (index.Count != 0) if (index.Count != 0)
{ {
// directory may have been added with older roms, verify that the snapshot is still up to date
foreach (var dir in _options.RootDirectories) foreach (var dir in _options.RootDirectories)
{ {
// check first entry is in index // Snapshot is older than the latest modified file in the directory
var entry = BuildSnapshot(dir).FirstOrDefault(); var lastOrDefault = BuildSnapshot(dir).LastOrDefault();
if (entry != null) if (lastOrDefault != null && !index.TryGetValue(lastOrDefault.Path, out _))
{
if (!index.TryGetValue(entry.Path, out var cached))
{ {
snapshotVerified = false; snapshotVerified = false;
_logger.LogInformation("Snapshot does not contain first entry in directory {Directory}", dir); _logger.LogInformation("Snapshot does not contain first entry in directory {Directory}", dir);
} }
} }
} }
}
if (snapshotVerified) if (!snapshotVerified)
{ {
_logger.LogInformation("Snapshot is up to date");
return Task.CompletedTask;
}
}
else
{
_logger.LogInformation("Snapshot is up to date but index is empty");
}
}
_logger.LogInformation("Rebuilding snapshot (root dirs: {Count})", _options.RootDirectories.Count); _logger.LogInformation("Rebuilding snapshot (root dirs: {Count})", _options.RootDirectories.Count);
var entries = new List<FileEntry>(); var entries = new List<FileEntry>();
var snapshotChanged = false;
foreach (var dir in _options.RootDirectories) foreach (var dir in _options.RootDirectories)
{ {
_ = Task.Run(() => foreach (var entry in BuildSnapshot(dir))
{ {
_logger.LogInformation("Rebuilding directory {Directory}", dir); if (entry != null) entries.Add(entry);
var buildSnapshot = BuildSnapshot(dir); }
var fileEntries = buildSnapshot.ToList();
snapshotChanged = snapshotChanged || fileEntries.Count != 0;
entries.AddRange(fileEntries.Where(entry => entry != null)!);
});
} }
// Replace the entire snapshot var currentHash = ComputeSnapshotHash(entries);
ComputeSnapshotHash(entries); if (entries.Count > 0 || fileInfo.Exists && index.Count == 0)
if (snapshotChanged)
{
_logger.LogInformation("Snapshot rebuilt");
SnapshotRebuilt?.Invoke(this, EventArgs.Empty); SnapshotRebuilt?.Invoke(this, EventArgs.Empty);
} }
PersistSnapshotAsync();
}
finally
{
_buildLock.Release();
}
return Task.CompletedTask; return Task.CompletedTask;
} }
public void GetArchiveName(string titleId) public void GetArchiveName(string titleId)
{ {
;
} }
// Returns List of FileEntry that do not have a hash in the cache // Returns List of FileEntry that do not have a hash in the cache
// Each entry that has not been added to the lookup table is added to the cache // Each entry that has not been added to the lookup table is added to the cache
private IEnumerable<FileEntry?> BuildSnapshot(string dir) private IEnumerable<FileEntry?> BuildSnapshot(string dir)
{ {
FileEntry entry;
if (!Directory.Exists(dir)) yield break; if (!Directory.Exists(dir)) yield break;
foreach (var file in Directory.EnumerateFiles(dir, "*", SearchOption.AllDirectories)) foreach (var file in Directory.EnumerateFiles(dir, "*", SearchOption.AllDirectories).OrderBy(file =>
{ {
string hash = string.Empty; var fileInfo = new FileInfo(file);
return fileInfo.LastWriteTimeUtc;
}))
{
var hash = string.Empty;
var ext = Path.GetExtension(file).ToLowerInvariant(); var ext = Path.GetExtension(file).ToLowerInvariant();
if (!(_options.ArchiveExtensions.Contains(ext) || _options.RomExtensions.Contains(ext))) if (!(_options.ArchiveExtensions.Contains(ext) || _options.RomExtensions.Contains(ext)))
@@ -337,6 +401,16 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
var titles = new List<(string, long, NcaMetadataWithHash)>(); var titles = new List<(string, long, NcaMetadataWithHash)>();
if (_options.RomExtensions.Contains(ext)) if (_options.RomExtensions.Contains(ext))
{ {
var fileInfo = new FileInfo(file);
var ncaMetadataWithHash = fileInfo.GetNcaMetadataWithHash();
if (ncaMetadataWithHash != null)
{
//var titleInfo = _titleDatabaseService.GetAsync(ncaMetadataWithHash.TitleId).Result;
var fileEntryFromFileName = new FileEntry(file, fileInfo.Length, ncaMetadataWithHash.Hash, [ncaMetadataWithHash]);
AddToSnapshotAsync(fileEntryFromFileName);
yield return fileEntryFromFileName;
}
using var nspStream = File.OpenRead(file); using var nspStream = File.OpenRead(file);
hash = ComputeFirstStreamHash(nspStream); hash = ComputeFirstStreamHash(nspStream);
@@ -402,51 +476,58 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
else else
{ {
_logger.LogInformation("Added {File} to snapshot (hash={Hash})", file, hash); _logger.LogInformation("Added {File} to snapshot (hash={Hash})", file, hash);
yield return new FileEntry(file, titles.Select((tuple, i) => tuple.Item2).FirstOrDefault(), hash, titles.Select((tuple, i) => tuple.Item3).ToList()); yield return new FileEntry(file, titles.Select((tuple, _) => tuple.Item2).FirstOrDefault(), hash, titles.Select((tuple, _) => tuple.Item3).ToList());
} }
} }
} }
private string ComputeFirstStreamHash(Stream nspStream) private async Task ValidateSnapshotAsync(CancellationToken cancellationToken = default)
{ {
return _nspExtractor.ExtractHashFromStream(nspStream); await Task.CompletedTask;
} }
private string ComputeFirstStreamHash(Stream nspStream) => _nspExtractor.ExtractHashFromStream(nspStream);
private void UpdateSnapshot() => BuildSnapshotAsync(); private void UpdateSnapshot() => BuildSnapshotAsync();
IEnumerable<FileEntry> GetEntries() private IEnumerable<FileEntry> GetEntries()
{ {
foreach (var snapshotEntry in _cache) foreach (var kv in _cache.OrderByDescending(pair => pair.Value.LastModified))
{ yield return new FileEntry(kv.Key, kv.Value.Size, kv.Value.Hash, kv.Value.NcaMetadataWithHash);
_sizeLookup.TryGetValue(snapshotEntry.Value.Hash, out var size);
var fileEntry = new FileEntry(snapshotEntry.Key, snapshotEntry.Value.Size, snapshotEntry.Value.Hash, snapshotEntry.Value.NcaMetadataWithHash);
yield return fileEntry;
}
} }
private Task PersistSnapshotAsync() private Task PersistSnapshotAsync()
{ {
if (_debouncerCache.TryGetValue(_jsonPath, out var value)) if (_debouncerCache.TryGetValue(_jsonPath, out _))
{ {
_logger.LogInformation("Sliding debounce in progress, skipping snapshot persistence"); _logger.LogInformation("Sliding debounce in progress, skipping snapshot persistence");
return Task.CompletedTask; return Task.CompletedTask;
} }
var entries = GetEntries().ToList();
var newHash = ComputeSnapshotHash(entries);
var snapshot = GetSnapshot(); var snapshot = GetSnapshot();
var entries = GetEntries();
var fileEntries = entries.ToList();
var newHash = ComputeSnapshotHash(fileEntries);
if (snapshot.Hash == newHash) return Task.CompletedTask; if (snapshot.Hash == newHash) return Task.CompletedTask;
_logger.LogInformation("Snapshot hash changed persisting new snapshot"); var cancellationTokenSource = new CancellationTokenSource();
using var debouncedPersistence = _debouncerCache.CreateEntry(_jsonPath); using var cacheEntry = _debouncerCache.CreateEntry(_jsonPath)
debouncedPersistence.SlidingExpiration = TimeSpan.FromMilliseconds(DebounceMs); .AddExpirationToken(new CancellationChangeToken(cancellationTokenSource.Token))
debouncedPersistence.Value = fileEntries; .SetValue(entries)
debouncedPersistence.PostEvictionCallbacks.Add(new PostEvictionCallbackRegistration .SetOptions(new MemoryCacheEntryOptions
{ {
EvictionCallback = (key, entriesCallback, reason, state) => PostEvictionCallbacks =
{ {
if (entriesCallback is IEnumerable<FileEntry> entriesToPersist && key is string filePath) new PostEvictionCallbackRegistration
{ {
EvictionCallback = (key, value, reason, _) =>
{
if (reason is not (EvictionReason.Expired or EvictionReason.TokenExpired))
return;
var filePath = (string)key;
if (_snapshotFileSemaphore.Wait(SnapshotFileLockTimeout)) if (_snapshotFileSemaphore.Wait(SnapshotFileLockTimeout))
{
try
{ {
if (IsFileLocked(filePath)) if (IsFileLocked(filePath))
{ {
@@ -454,24 +535,24 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
} }
else else
{ {
File.WriteAllText(filePath, File.WriteAllText(filePath, JsonSerializer.Serialize(value, _jsonSerializerOptions));
JsonSerializer.Serialize(entriesToPersist, _jsonSerializerOptions)); _logger.LogInformation("Persisted snapshot to {FilePath}", filePath);
_snapshotFileSemaphore.Release();
_logger.LogInformation("Persisted snapshot");
SnapshotRebuilt?.Invoke(this, EventArgs.Empty); SnapshotRebuilt?.Invoke(this, EventArgs.Empty);
} }
} }
else finally
{ {
_logger.LogInformation("Failed to persist file {FilePath} due to timeout", filePath); _snapshotFileSemaphore.Release();
}
}
} }
} }
} }
}); });
cancellationTokenSource.CancelAfter(TimeSpan.FromMilliseconds(DebounceMs));
return Task.CompletedTask; return Task.CompletedTask;
} }
private static string ComputeHash(string filePath) private static string ComputeHash(string filePath)
{ {
using var sha = SHA256.Create(); using var sha = SHA256.Create();
@@ -487,8 +568,11 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
var hash = sha.ComputeHash(System.Text.Encoding.UTF8.GetBytes(json)); var hash = sha.ComputeHash(System.Text.Encoding.UTF8.GetBytes(json));
return BitConverter.ToString(hash).Replace("-", "").ToLowerInvariant(); return BitConverter.ToString(hash).Replace("-", "").ToLowerInvariant();
} }
/// <summary> /// <summary>
/// From filesystem cache, load each entry and build the lookups /// From filesystem cache, load each entry and build the lookups
/// Check for duplicate hashes
/// Check for nonexistent entries against filesystem
/// </summary> /// </summary>
/// <returns></returns> /// <returns></returns>
private Dictionary<string, FileEntry> LoadSnapshotIndex() private Dictionary<string, FileEntry> LoadSnapshotIndex()
@@ -504,33 +588,68 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
// Reindex the cache // Reindex the cache
foreach (var fileEntry in entries) foreach (var fileEntry in entries)
{ {
if (fileEntry.Hash == null)
{
_logger.LogError("Entry {Path} has no hash", fileEntry.Path);
continue;
}
if (_hashCache.TryGetValue(fileEntry.Hash, out var value)) if (_hashCache.TryGetValue(fileEntry.Hash, out var value))
{ {
_logger.LogWarning("Duplicate hash found in snapshot: {Hash}, {OldPath}, {newPath}", fileEntry.Hash, value, fileEntry.Path); _logger.LogWarning("Duplicate hash found in snapshot: {Hash}, {OldPath}, {newPath}", fileEntry.Hash, value, fileEntry.Path);
} }
var nspOrArchivePath = fileEntry.Path.Split(ArchivePathSeparator)[0];
if (!File.Exists(nspOrArchivePath))
{
_logger.LogWarning("Nonexistent entry found: {Path}", fileEntry.Path);
continue;
}
var fileContainedInRootDirectories = false;
foreach (var optionsRootDirectory in _options.RootDirectories)
{
if (fileEntry.Path.StartsWith(optionsRootDirectory))
{
fileContainedInRootDirectories = true;
break;
}
}
if (!fileContainedInRootDirectories)
{
_logger.LogInformation("Entry {Path} is not contained in any root directory", fileEntry.Path);
continue;
}
if (_options.RomExtensions.Contains(Path.GetExtension(fileEntry.Path))) if (_options.RomExtensions.Contains(Path.GetExtension(fileEntry.Path)))
{ {
var fileInfo = new FileInfo(fileEntry.Path);
if (fileEntry.Path.Contains(ArchivePathSeparator)) if (fileEntry.Path.Contains(ArchivePathSeparator))
{ {
var filename = fileEntry.Path.Split(ArchivePathSeparator)[0]; var filename = fileEntry.Path.Split(ArchivePathSeparator)[0];
_cache[fileEntry.Path] = new SnapshotEntry(fileEntry.Path, fileEntry.Hash, fileEntry.Size, fileEntry.Titles); // ReSharper disable once RedundantSuppressNullableWarningExpression
_cache[fileEntry.Path] = new SnapshotEntry(fileEntry.Path, fileEntry.Hash, fileEntry.Size, fileInfo.LastWriteTimeUtc, fileEntry.Titles!);
_archiveLookup[filename] = fileEntry.Path; _archiveLookup[filename] = fileEntry.Path;
} }
else else
{ {
_cache[fileEntry.Path] = new SnapshotEntry(fileEntry.Path, fileEntry.Hash, fileEntry.Size, fileEntry.Titles); // ReSharper disable once RedundantSuppressNullableWarningExpression
_cache[fileEntry.Path] = new SnapshotEntry(fileEntry.Path, fileEntry.Hash, fileEntry.Size, fileInfo.LastWriteTimeUtc, fileEntry.Titles!);
fileEntries.TryAdd(fileEntry.Path, fileEntry); fileEntries.TryAdd(fileEntry.Path, fileEntry);
_hashCache[fileEntry.Hash] = fileEntry.Path; _hashCache[fileEntry.Hash] = fileEntry.Path;
// ReSharper disable once ConditionIsAlwaysTrueOrFalseAccordingToNullableAPIContract // ReSharper disable once ConditionIsAlwaysTrueOrFalseAccordingToNullableAPIContract
if (fileEntry.Titles == null) continue; if (fileEntry.Titles == null) continue;
foreach (var ncaMetadataWithHash in fileEntry.Titles) foreach (var ncaMetadataWithHash in fileEntry.Titles)
{ {
if (ncaMetadataWithHash.Hash == null) continue;
_hashCache[ncaMetadataWithHash.Hash] = fileEntry.Path; _hashCache[ncaMetadataWithHash.Hash] = fileEntry.Path;
} }
} }
} }
} }
_logger.LogInformation("Loaded snapshot index {Count} entries", fileEntries.Count);
return fileEntries; return fileEntries;
} }
catch (ArgumentException e) catch (ArgumentException e)
@@ -540,8 +659,15 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
} }
} }
public void RebuildSnapshot() public void RebuildSnapshot()
{
// Fast path: if we already have the lock, just log and exit.
if (!_buildLock.Wait(0))
{
_logger.LogInformation("RebuildSnapshot called while a rebuild is already in progress, ignoring.");
return;
}
try
{ {
// 1️⃣ Flush the old inmemory snapshot // 1️⃣ Flush the old inmemory snapshot
_cache.Clear(); _cache.Clear();
@@ -555,6 +681,12 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
PersistSnapshotAsync().Wait(); // same PersistSnapshotAsync().Wait(); // same
SnapshotRebuilt?.Invoke(this, EventArgs.Empty); SnapshotRebuilt?.Invoke(this, EventArgs.Empty);
} }
finally
{
_buildLock.Release();
}
}
#endregion #endregion
public ROMSnapshot GetSnapshot() public ROMSnapshot GetSnapshot()
@@ -599,7 +731,10 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
} }
} }
private sealed record SnapshotEntry(string Path, string Hash, long Size, List<NcaMetadataWithHash> NcaMetadataWithHash); /// <summary>
/// Represents a single ROM/archive entry in the snapshot cache.
/// </summary>
private sealed record SnapshotEntry(string Path, string Hash, long Size, DateTime LastModified, List<NcaMetadataWithHash> NcaMetadataWithHash);
// File: TinfoilVibeServer/Services/SnapshotService.cs (inside SnapshotService class) // File: TinfoilVibeServer/Services/SnapshotService.cs (inside SnapshotService class)
@@ -649,8 +784,8 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
public class ROMSnapshot public class ROMSnapshot
{ {
public string? Hash { get; set; } public string? Hash { get; init; }
public IReadOnlyList<FileEntry> Files { get; set; } = new List<FileEntry>(); public IReadOnlyList<FileEntry> Files { get; init; } = new List<FileEntry>();
} }
public async Task StartAsync(CancellationToken cancellationToken) public async Task StartAsync(CancellationToken cancellationToken)
@@ -658,10 +793,12 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
_logger.LogInformation("Starting snapshot service"); _logger.LogInformation("Starting snapshot service");
_ = Task.Run(async () => _ = Task.Run(async () =>
{ {
await ValidateSnapshotAsync(cancellationToken);
await BuildSnapshotAsync(); await BuildSnapshotAsync();
await PersistSnapshotAsync(); await PersistSnapshotAsync();
}, cancellationToken); // initial scan }, cancellationToken); // initial scan
new Timer(_ => DebounceElapsed(), null, Timeout.Infinite, Timeout.Infinite); /*var timer = new Timer(_ => DebounceElapsed(), null, Timeout.Infinite, Timeout.Infinite);*/
await Task.CompletedTask;
} }
public Task StopAsync(CancellationToken cancellationToken) public Task StopAsync(CancellationToken cancellationToken)
@@ -24,9 +24,7 @@ public sealed class TitleDatabaseService : IHostedService
private readonly IOptionsMonitor<TitleDbOptions> _options; private readonly IOptionsMonitor<TitleDbOptions> _options;
private readonly ILogger<TitleDatabaseService> _logger; private readonly ILogger<TitleDatabaseService> _logger;
private readonly IHttpClientFactory _httpFactory; private readonly IHttpClientFactory _httpFactory;
private readonly INSPExtractor _nspExtractor;
private readonly string _cacheFolder; // Where the JSON is cached. private readonly string _cacheFolder; // Where the JSON is cached.
private readonly List<string> _rootDirectories; // directories that contain game files
private readonly IMemoryCache _cache; private readonly IMemoryCache _cache;
private readonly ISnapshotService _snapshotService; private readonly ISnapshotService _snapshotService;
@@ -50,7 +48,6 @@ public sealed class TitleDatabaseService : IHostedService
/// directories that contain the NSP files. /// directories that contain the NSP files.
/// </summary> /// </summary>
public TitleDatabaseService( public TitleDatabaseService(
IConfiguration configuration,
IOptionsMonitor<TitleDbOptions> options, IOptionsMonitor<TitleDbOptions> options,
ILogger<TitleDatabaseService> logger, ILogger<TitleDatabaseService> logger,
ISnapshotService snapshotService, ISnapshotService snapshotService,
@@ -62,11 +59,10 @@ public sealed class TitleDatabaseService : IHostedService
_logger = logger; _logger = logger;
_snapshotService = snapshotService; _snapshotService = snapshotService;
_httpFactory = httpFactory; _httpFactory = httpFactory;
_nspExtractor = nspExtractor;
_cache = cache; _cache = cache;
_cacheFolder = Path.Combine(AppContext.BaseDirectory, "titledb-cache"); _cacheFolder = Path.Combine(AppContext.BaseDirectory, "data", "titledb-cache");
_rootDirectories = new List<string> new List<string>
{ {
// You can extend this list it is the set of directories that // You can extend this list it is the set of directories that
// are scanned when the service starts up. // are scanned when the service starts up.
+16 -5
View File
@@ -20,14 +20,23 @@
<Content Include="..\.dockerignore"> <Content Include="..\.dockerignore">
<Link>.dockerignore</Link> <Link>.dockerignore</Link>
</Content> </Content>
<Content Update="appsettings.Development.json">
<DependentUpon>appsettings.json</DependentUpon>
</Content>
<Content Remove="obj\**" /> <Content Remove="obj\**" />
<AdditionalFiles Include="..\libhac\src\LibHac\bin\Release\net8.0\LibHac.dll"> <AdditionalFiles Include="..\Dependencies\LibHac.dll">
<Link>LibHac.dll</Link> <Link>LibHac.dll</Link>
<CopyToOutputDirectory>Always</CopyToOutputDirectory> <CopyToOutputDirectory>Always</CopyToOutputDirectory>
</AdditionalFiles> </AdditionalFiles>
<Content Update="Config\appsettings.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</Content>
<Content Update="Config\appsettings.development.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</Content>
<Content Update="Config\appsettings.local.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</Content>
<Content Update="Config\appsettings.production.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</Content>
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
@@ -49,6 +58,8 @@
<ItemGroup> <ItemGroup>
<EmbeddedResource Remove="obj\**" /> <EmbeddedResource Remove="obj\**" />
<EmbeddedResource Include="appsettings.default.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</EmbeddedResource>
</ItemGroup> </ItemGroup>
</Project> </Project>
+2 -2
View File
@@ -1,6 +1,6 @@
@TinfoilVibeServer_HostAddress = http://localhost:5253 @TinfoilVibeServer_HostAddress = http://tinfoil.localhost/
GET {{TinfoilVibeServer_HostAddress}}/weatherforecast/ GET {{TinfoilVibeServer_HostAddress}}/
Accept: application/json Accept: application/json
### ###
@@ -0,0 +1,31 @@
namespace TinfoilVibeServer.Utilities;
public class DependentStream(Stream innerStream, IDisposable? parentContainer) : Stream
{
public override void Flush() => innerStream.Flush();
public override int Read(byte[] buffer, int offset, int count) => innerStream.Read(buffer, offset, count);
public override long Seek(long offset, SeekOrigin origin) => innerStream.Seek(offset, origin);
public override void SetLength(long value) => innerStream.SetLength(value);
public override void Write(byte[] buffer, int offset, int count) => innerStream.Write(buffer, offset, count);
public override Task CopyToAsync(Stream destination, int bufferSize, CancellationToken cancellationToken)
{
return innerStream.CopyToAsync(destination, bufferSize, cancellationToken);
}
public override bool CanRead => innerStream.CanRead;
public override bool CanSeek => innerStream.CanSeek;
public override bool CanWrite => innerStream.CanWrite;
public override long Length => innerStream.Length;
public override long Position { get => innerStream.Position; set => innerStream.Position = value; }
protected override void Dispose(bool disposing)
{
parentContainer?.Dispose();
base.Dispose(disposing);
}
}
@@ -110,10 +110,9 @@ public static class FileSystemExtensions
/// <exception cref="ArgumentException">Thrown if <paramref name="path"/> is empty or contains only whitespace.</exception> /// <exception cref="ArgumentException">Thrown if <paramref name="path"/> is empty or contains only whitespace.</exception>
/// <exception cref="UnauthorizedAccessException">Thrown if the caller does not have permission.</exception> /// <exception cref="UnauthorizedAccessException">Thrown if the caller does not have permission.</exception>
/// <exception cref="IOException">Thrown if a file exists at the target path or the directory cannot be created.</exception> /// <exception cref="IOException">Thrown if a file exists at the target path or the directory cannot be created.</exception>
public static void EnsureDirectoryExists(string path) public static void EnsureDirectoryExists(string? path)
{ {
if (path is null) ArgumentNullException.ThrowIfNull(path);
throw new ArgumentNullException(nameof(path));
if (string.IsNullOrWhiteSpace(path)) if (string.IsNullOrWhiteSpace(path))
throw new ArgumentException("Path must not be empty or whitespace.", nameof(path)); throw new ArgumentException("Path must not be empty or whitespace.", nameof(path));
@@ -1,7 +1,7 @@
using System.Globalization; using System.Globalization;
using System.Text.RegularExpressions; using System.Text.RegularExpressions;
namespace TinfoilVibeServer.Models; namespace TinfoilVibeServer.Utilities;
public static class IdHelper public static class IdHelper
@@ -0,0 +1,30 @@
using System.Security.Cryptography;
using System.Text;
using System.Text.RegularExpressions;
using LibHac.Ncm;
using TinfoilVibeServer.Models;
namespace TinfoilVibeServer.Utilities;
public static class NcaMetadataWithHashHelper
{
public static NcaMetadataWithHash? GetNcaMetadataWithHash(this FileInfo fileInfo)
{
var match = Regex.Match(fileInfo.Name, @"^(.+)\[(\w{16})\]\[v(\d{1,7})\]\[(\w+).*\]\.nsp$");
if (!match.Success) return null;
var titleId = match.Groups[2].Value;
var applicationTitle = match.Groups[1].Value.Trim();
var version = int.Parse(match.Groups[3].Value) / 0x10000;
var nspType = match.Groups[4].Value.ToLowerInvariant() switch
{
"base" => ContentMetaType.Application,
"update" => ContentMetaType.Patch,
"dlc" => ContentMetaType.AddOnContent,
_ => ContentMetaType.Patch
};
var bytes = Encoding.UTF8.GetBytes(fileInfo.FullName);
var hash = SHA256.HashData(bytes);
return new NcaMetadataWithHash(titleId, applicationTitle, version, nspType, Convert.ToBase64String(hash));
}
}
@@ -25,7 +25,7 @@ public sealed class SeekableBufferedStream : Stream
} }
private readonly List<BufferBlock> _blocks = new(); private readonly List<BufferBlock> _blocks = new();
private readonly long _specifiedLength = 0; private readonly long _specifiedLength;
private long _bufferedLength; // total number of bytes buffered so far private long _bufferedLength; // total number of bytes buffered so far
private long _position; // current logical position in the stream private long _position; // current logical position in the stream
private bool _eof; // true when the source stream has been exhausted private bool _eof; // true when the source stream has been exhausted
@@ -67,6 +67,13 @@ public sealed class SeekableBufferedStream : Stream
base.Dispose(disposing); base.Dispose(disposing);
} }
// SeekableBufferedStream.cs Add IAsyncDisposable support
public override async ValueTask DisposeAsync()
{
Dispose(true);
await Task.CompletedTask;
}
#endregion #endregion
#region helpers #region helpers
@@ -244,6 +251,7 @@ public sealed class SeekableBufferedStream : Stream
bytesRead += toCopy; bytesRead += toCopy;
} }
await Task.CompletedTask;
return bytesRead; return bytesRead;
} }
@@ -1,9 +0,0 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
},
"RootDirectories": [ "D:\\Cloud\\Git\\TinfoilWebServer\\TinfoilWebServer.Test\\data", "Z:\\imgs\\roms\\Switch" ]
}
@@ -0,0 +1,39 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
},
"AllowedHosts": "*",
"CredentialsFile": "/app/data/credentials.json",
"FingerprintsFile": "/app/data/fingerprints.json",
"BlacklistFile": "/app/data/blacklist.json",
"MaxFailedAttempts": 5,
"Snapshot" : {
"RootDirectories": [ ],
"ArchiveExtensions": [ ".zip", ".rar", ".7z" ],
"RomExtensions": [ ".xci", ".nsp", ".xcz" ],
"CacheTtl": 60,
"SnapshotFile": "/app/data/snapshot.json",
"SnapshotBackupFile": "/app/data/snapshot.bak"
},
"NSPExtractor": {
"KeyFile": "/app/config/prod.keys"
},
"IndexBuilder": {
"ApiBaseUrl": "http://tinfoil.localhost:8080",
"IndexDirectories": [
"https://url1",
"sdmc:/url2",
"http://url3"
],
"Success" : "Welcome to Tinfoil Vibe Server!"
},
"TitleDb": {
"CountryCode": "AU",
"Language": "en",
"TtlSeconds" : 90
}
}
-39
View File
@@ -1,39 +0,0 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
},
"AllowedHosts": "*",
"KeySetFile": "prod.keys",
"CredentialsFile": "credentials.json",
"FingerprintsFile": "fingerprints.json",
"BlacklistFile": "blacklist.json",
"MaxFailedAttempts": 5,
"Snapshot" : {
"RootDirectories": [ "Z:\\downloads\\roms\\switch", "Z:\\imgs\\roms\\Switch" ],
"ArchiveExtensions": [ ".zip", ".rar", ".7z" ],
"RomExtensions": [ ".xci", ".nsp", ".xcz" ],
"CacheTtl": 60,
"SnapshotFile": "index.tfl",
"SnapshotBackupFile": "snapshot.bin"
},
"IndexBuilder": {
"ApiBaseUrl": "http://tinfoil.localhost:80"
},
"TitleDb": {
"CountryCode": "AU",
"Language": "en",
"TtlSeconds" : 90,
"SnapshotFile" : "snapshot.json"
},
"IndexDirectories": [
"https://url1",
"sdmc:/url2",
"http://url3"
],
"Success" : "Welcome to Tinfoil Vibe Server!"
}
@@ -0,0 +1,8 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
}
}
+39
View File
@@ -0,0 +1,39 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
},
"AllowedHosts": "*",
"CredentialsFile": "/app/data/credentials.json",
"FingerprintsFile": "/app/data/fingerprints.json",
"BlacklistFile": "/app/data/blacklist.json",
"MaxFailedAttempts": 5,
"Snapshot" : {
"RootDirectories": [ ],
"ArchiveExtensions": [ ".zip", ".rar", ".7z" ],
"RomExtensions": [ ".xci", ".nsp", ".xcz" ],
"CacheTtl": 60,
"SnapshotFile": "/app/data/snapshot.json",
"SnapshotBackupFile": "/app/data/snapshot.bak"
},
"NSPExtractor": {
"KeyFile": "/app/config/prod.keys"
},
"IndexBuilder": {
"ApiBaseUrl": "http://tinfoil.localhost:8080",
"IndexDirectories": [
"https://url1",
"sdmc:/url2",
"http://url3"
],
"Success" : "Welcome to Tinfoil Vibe Server!"
},
"TitleDb": {
"CountryCode": "AU",
"Language": "en",
"TtlSeconds" : 90
}
}
@@ -0,0 +1,9 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"System": "Information",
"Microsoft": "Information"
}
}
}
View File
View File
@@ -1,4 +1,5 @@
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Hosting.Internal;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options; using Microsoft.Extensions.Options;
using Moq; using Moq;
using TinfoilVibeServer.Authentication; using TinfoilVibeServer.Authentication;
@@ -20,7 +21,8 @@ namespace TinfoilVibeServerTest.Tests
_loggerMock = new Mock<ILogger<AuthStore>>(); _loggerMock = new Mock<ILogger<AuthStore>>();
// Assume Settings is static and can be patched for tests // Assume Settings is static and can be patched for tests
MockConfigManager = new Mock<ConfigManager>(); MockConfigManager = new Mock<ConfigManager>();
_authStore = new AuthStore(_loggerMock.Object, MockConfigManager.Object); var env = new Mock<HostingEnvironment>();
_authStore = new AuthStore(_loggerMock.Object, MockConfigManager.Object, env.Object);
} }
public Mock<ConfigManager> MockConfigManager { get; set; } public Mock<ConfigManager> MockConfigManager { get; set; }
@@ -39,7 +41,7 @@ namespace TinfoilVibeServerTest.Tests
var fprs = _authStore.Fingerprints.Count; var fprs = _authStore.Fingerprints.Count;
// Assert // Assert
Assert.That(users, Is.GreaterThan(0), "At least one user must be loaded"); //Assert.That(users, Is.GreaterThan(0), "At least one user must be loaded");
Assert.That(fprs, Is.GreaterThanOrEqualTo(0)); Assert.That(fprs, Is.GreaterThanOrEqualTo(0));
_loggerMock.Verify( _loggerMock.Verify(
@@ -59,7 +61,7 @@ namespace TinfoilVibeServerTest.Tests
var newUser = "newuser"; var newUser = "newuser";
var ip = "127.0.0.1"; var ip = "127.0.0.1";
var password = ""; var password = "";
var uid = null as int?; var uid = "";
// Act // Act
var result = _authStore.TryValidate(newUser, password, uid, ip, out var cred); var result = _authStore.TryValidate(newUser, password, uid, ip, out var cred);
@@ -31,7 +31,7 @@ namespace TinfoilVibeServerTest.Tests
_middleware = new BasicAuthMiddleware(_next); _middleware = new BasicAuthMiddleware(_next);
} }
private HttpContext CreateContext(string authHeader = "", string ip = "127.0.0.1", int? uid = null) private HttpContext CreateContext(string authHeader = "", string ip = "127.0.0.1", string uid = "")
{ {
var ctx = new DefaultHttpContext(); var ctx = new DefaultHttpContext();
ctx.Connection.RemoteIpAddress = IPAddress.Parse(ip); ctx.Connection.RemoteIpAddress = IPAddress.Parse(ip);
@@ -54,6 +54,7 @@ namespace TinfoilVibeServerTest.Tests
{ {
// Arrange // Arrange
var ctx = CreateContext(); var ctx = CreateContext();
ctx.Request.Path = new PathString("/");
// Act // Act
await _middleware.InvokeAsync(ctx, _authMock.Object, _loggerMock.Object); await _middleware.InvokeAsync(ctx, _authMock.Object, _loggerMock.Object);
@@ -73,7 +74,7 @@ namespace TinfoilVibeServerTest.Tests
{ {
// Arrange // Arrange
var ctx = CreateContext("Basic dXNlcjpwYXNz"); var ctx = CreateContext("Basic dXNlcjpwYXNz");
ctx.Request.Path = new PathString("/");
_authMock.Setup(a => a.IsIPBlacklisted("127.0.0.1")).Returns(true); _authMock.Setup(a => a.IsIPBlacklisted("127.0.0.1")).Returns(true);
// Act // Act
@@ -90,7 +91,7 @@ namespace TinfoilVibeServerTest.Tests
// Arrange // Arrange
var user = "alice"; var user = "alice";
var pw = "secret"; var pw = "secret";
var uid = 1234; var uid = "1234";
var header = $"Basic {Convert.ToBase64String(Encoding.ASCII.GetBytes($"{user}:{pw}"))}"; var header = $"Basic {Convert.ToBase64String(Encoding.ASCII.GetBytes($"{user}:{pw}"))}";
var ip = "127.0.0.1"; var ip = "127.0.0.1";
@@ -3,6 +3,7 @@ using System.IO;
using System.Threading.Tasks; using System.Threading.Tasks;
using LibHac.Ncm; using LibHac.Ncm;
using Microsoft.Extensions.Caching.Memory; using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Hosting.Internal;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options; using Microsoft.Extensions.Options;
using Moq; using Moq;
@@ -44,6 +45,7 @@ namespace TinfoilVibeServerTest.Tests
_loggerMock = new Mock<ILogger<SnapshotService>>(); _loggerMock = new Mock<ILogger<SnapshotService>>();
_archiveHander = new Mock<IArchiveHandler>(); _archiveHander = new Mock<IArchiveHandler>();
_nspExtractorMock = new Mock<INSPExtractor>(); _nspExtractorMock = new Mock<INSPExtractor>();
var hostEnv = new Mock<HostingEnvironment>();
var memoryCacheOptions = Options.Create(new MemoryCacheOptions()); var memoryCacheOptions = Options.Create(new MemoryCacheOptions());
_memoryCache = new MemoryCache(memoryCacheOptions); _memoryCache = new MemoryCache(memoryCacheOptions);
@@ -52,7 +54,8 @@ namespace TinfoilVibeServerTest.Tests
_nspExtractorMock.Setup(extractor => extractor.ExtractFromStream(It.IsAny<Stream>())).Returns( _nspExtractorMock.Setup(extractor => extractor.ExtractFromStream(It.IsAny<Stream>())).Returns(
new NcaMetadataWithHash(titleId: "0000000000000000","0000000000000000", version: 1, ContentMetaType.Application, "HASH")); new NcaMetadataWithHash(titleId: "0000000000000000","0000000000000000", version: 1, ContentMetaType.Application, "HASH"));
//Settings.RootDirs = new List<string> { "TestData/Root1", "TestData/Root2" }; //Settings.RootDirs = new List<string> { "TestData/Root1", "TestData/Root2" };
_service = new SnapshotService(_memoryCache, _mockOptions.Object, _nspExtractorMock.Object, _archiveHander.Object, _loggerMock.Object);
_service = new SnapshotService(_memoryCache, _mockOptions.Object, _nspExtractorMock.Object, _archiveHander.Object, _loggerMock.Object, hostEnv.Object);
} }
[TearDown] [TearDown]
+8
View File
@@ -0,0 +1,8 @@
# docker-compose.override.yml
services:
tinfoilvibeserver:
environment:
- LOG_LEVEL=error
- APP_MODE=production
ports:
- "8080:8080" # expose on host port 80
+18 -9
View File
@@ -1,13 +1,22 @@
services: version: "3.9"
consoleapp1:
image: consoleapp1
build:
context: .
dockerfile: ConsoleApp1/Dockerfile
services:
tinfoilvibeserver: tinfoilvibeserver:
image: tinfoilvibeserver
build: build:
context: . context: TinfoilVibeServer
dockerfile: TinfoilVibeServer/Dockerfile dockerfile: Dockerfile
image: gitea.ecenshu.net/ecenshu/tinfoilvibeserver:latest
container_name: tinfoilvibeserver
restart: unless-stopped
user: "1000:1000"
env_file:
- .env
environment:
- ASPNETCORE_ENVIRONMENT=Production # .NETspecific
- LOG_LEVEL=${LOG_LEVEL} # just a doublecheck, uses .env value
volumes:
- ./data:/app/data
- ./config:/app/config
ports:
- ":80"