24 Commits

Author SHA1 Message Date
ecenshu bea70381dd Less strict uri validator
ci / build_linux (pull_request) Failing after 14m4s
ci / build_linux (push) Failing after 14m25s
Check against archive name or nsp instead of fileentry.path
2025-11-15 19:16:35 +10:30
ecenshu 91f394f81b If filename can extract to a NcaMetadata entry, don't use nspextractor to pull information
ci / build_linux (push) Successful in 6m53s
ci / build_linux (pull_request) Successful in 3m25s
Scan directories sequentially to reduce memory footprint
2025-11-15 17:28:41 +10:30
ecenshu 17d12fef6a Use a resource to initialise appsettings.json in config folder (#2)
Build & Push Docker image / build-and-push (push) Successful in 8m18s
ci / build_linux (push) Successful in 3m36s
Watch for KeySet, initial install will not have a valid value
TitleDatabase will use data folder

Reviewed-on: #2
Co-authored-by: Huy Nguyen <ecenshu@gmail.com>
Co-committed-by: Huy Nguyen <ecenshu@gmail.com>
2025-11-13 23:52:34 +00:00
ecenshu 97fc2dc872 Use sanitized github.ref
Build & Push Docker image / build-and-push (push) Successful in 15m11s
ci / build_linux (push) Successful in 5m29s
2025-11-13 19:44:36 +10:30
ecenshu a1ea34bc01 feature/ci (#1)
Build & Push Docker image / build-and-push (push) Has been cancelled
ci / build_linux (push) Has been cancelled
Consolidate data and config into separate folders that will be expected to be mapped in the container

Reviewed-on: #1
Co-authored-by: Huy Nguyen <ecenshu@gmail.com>
Co-committed-by: Huy Nguyen <ecenshu@gmail.com>
2025-11-13 09:11:21 +00:00
ecenshu 314af37b3d Workflow cache
Build & Push Docker image / build-and-push (push) Successful in 12m21s
ci / build_linux (push) Successful in 5m42s
2025-11-08 10:23:46 +10:30
ecenshu 877db194c2 tabs
Build & Push Docker image / build-and-push (push) Has been cancelled
ci / build_linux (push) Has been cancelled
2025-11-08 10:22:23 +10:30
ecenshu c4c15dbada workflow cache
Build & Push Docker image / build-and-push (push) Has been cancelled
2025-11-08 10:19:58 +10:30
ecenshu 301525e198 dots...
Build & Push Docker image / build-and-push (push) Waiting to run
ci / build_linux (push) Has been cancelled
2025-11-07 23:14:43 +10:30
ecenshu fa8f0d7443 LibHac.dll from Dependencies
Build & Push Docker image / build-and-push (push) Failing after 5m38s
ci / build_linux (push) Has been cancelled
2025-11-07 22:47:48 +10:30
ecenshu 1fdef1bcc8 Merge branch 'feature/gpt-oss-01'
Build & Push Docker image / build-and-push (push) Failing after 6m59s
ci / build_linux (push) Failing after 7m44s
2025-11-07 22:33:10 +10:30
ecenshu 53ba636258 At end maybe?
Build & Push Docker image / build-and-push (push) Failing after 5m14s
ci / build_linux (push) Has been cancelled
2025-11-07 22:23:45 +10:30
ecenshu a25f5f602e Hardcodepath
Build & Push Docker image / build-and-push (push) Failing after 1m50s
ci / build_linux (push) Has been cancelled
2025-11-07 22:17:31 +10:30
ecenshu b9370eb2d5 pathing problem?
Build & Push Docker image / build-and-push (push) Failing after 1m49s
ci / build_linux (push) Failing after 1m38s
2025-11-07 22:06:25 +10:30
ecenshu 7c6fba9b3f More copy pasta
Build & Push Docker image / build-and-push (push) Failing after 1m50s
ci / build_linux (push) Failing after 1m56s
2025-11-07 21:57:44 +10:30
ecenshu 4eb8324056 Vars instead of env
Build & Push Docker image / build-and-push (push) Failing after 1m49s
ci / build_linux (push) Failing after 1m34s
2025-11-07 21:50:27 +10:30
ecenshu 6cb78c91fa Use ENV for tags instead of SECRETS
Build & Push Docker image / build-and-push (push) Failing after 1m54s
ci / build_linux (push) Failing after 1m43s
2025-11-07 21:46:04 +10:30
ecenshu a9184acd23 Shell to lowercase
Build & Push Docker image / build-and-push (push) Failing after 1m57s
ci / build_linux (push) Failing after 1m32s
2025-11-07 21:40:17 +10:30
ecenshu fae1979e04 Tabs
Build & Push Docker image / build-and-push (push) Failing after 40s
ci / build_linux (push) Failing after 1m13s
2025-11-07 21:31:19 +10:30
ecenshu 36f2f0c2f9 BuildX doesn't like camelcase tags 2025-11-07 21:24:12 +10:30
ecenshu 35d4eccdfd Remove demo
Build & Push Docker image / build-and-push (push) Failing after 1m46s
ci / build_linux (push) Failing after 1m32s
Correct the tags to align with Gitea Repo expected tags
2025-11-07 21:19:38 +10:30
ecenshu a81f67536f ci copy pasta error
Gitea Actions Demo / Explore-Gitea-Actions (push) Has been cancelled
Build & Push Docker image / build-and-push (push) Failing after 1m55s
ci / build_linux (push) Failing after 1m32s
2025-11-07 20:50:20 +10:30
ecenshu ce68860175 First attempt with workflows
ci / tsan (C++17, tsan) (push) Waiting to run
ci / ubsan (C++17, ubsan) (push) Waiting to run
ci / results (push) Blocked by required conditions
Build & Push Docker image / build-and-push (push) Has been cancelled
ci / g++11.3.0 (C++17, Debug) (push) Has been cancelled
ci / g++12.3.0 (C++20, Debug) (push) Has been cancelled
ci / g++13.3.0 (C++20, Debug) (push) Has been cancelled
ci / g++14.3.0 (C++20, Debug) (push) Has been cancelled
ci / g++9.4.0 (C++17, Debug) (push) Has been cancelled
Gitea Actions Demo / Explore-Gitea-Actions (push) Has been cancelled
ci / clang20 (C++17, Debug) (push) Failing after 3m29s
ci / g++10.3.0 (C++17, Debug) (push) Has been cancelled
ci / clang15 (C++17, Debug) (push) Failing after 54s
ci / clang17 (C++17, Debug) (push) Failing after 54s
ci / clang18 (C++17, Debug) (push) Failing after 52s
ci / clang19 (C++17, Debug) (push) Failing after 53s
ci / g++15 (C++17, Debug) (push) Failing after 50s
ci / clang20 (C++17, Release) (push) Failing after 52s
ci / g++15 (C++17, Release) (push) Failing after 51s
ci / asan (C++17, asan) (push) Failing after 50s
2025-11-07 20:43:12 +10:30
ecenshu becc41a5f0 Testing actions
Gitea Actions Demo / Explore-Gitea-Actions (push) Successful in 2m29s
2025-11-04 08:35:55 +10:30
45 changed files with 957 additions and 412 deletions
+2 -1
View File
@@ -22,4 +22,5 @@
**/secrets.dev.yaml
**/values.dev.yaml
LICENSE
README.md
README.md
data/
+136
View File
@@ -0,0 +1,136 @@
name: Build & Push Docker image
on:
# Trigger on pushes to main or release branches, and on manual workflow dispatch
push:
branches:
- main
- 'release/**'
- 'beta/**'
workflow_dispatch:
jobs:
build-and-push:
runs-on: ubuntu-latest
environment: production
steps:
- name: Convert to lowercase
id: github_repository_to_lowercase
run: |
# Grab the value (you can also use `${{ github.ref }}`, `${{ secrets.MY_SECRET }}`, etc.)
raw_value="${{ github.repository }}"
# Convert to lower case
lower_value=$(echo "$raw_value" | tr '[:upper:]' '[:lower:]')
# Export it to the workflow environment
# echo "MY_LOWER=$lower_value" >> $GITHUB_ENV
# If you want to use it as an output of this step:
echo "lowercase=$lower_value" >> $GITHUB_OUTPUT
- name: Convert ref to buildx safe value
id: docker_tag_from_ref
shell: bash
run: |
# Grab the raw ref
REF="${{ github.ref }}"
# Strip the "refs/*/" prefix (refs/heads/, refs/tags/…)
TAG=${REF#refs/*/}
# Replace characters that Docker tags disallow
# * "/" → "-"
# * ":" → "-"
# * Any other nonalphanumeric / . / _ / - → "-"
TAG=${TAG//\//-}
TAG=${TAG//:/-}
TAG=${TAG//[^a-zA-Z0-9._-]/-}
# (Optional) force lowercase Docker tags are casesensitive,
# but many people prefer lowercase
TAG=${TAG,,}
# Export to the action's output
echo "docker-tag=${TAG}" >> $GITHUB_OUTPUT
# ------------------------------------------------------------------
# 1. Checkout repository
# ------------------------------------------------------------------
- name: Checkout source
uses: actions/checkout@v4
with:
fetch-depth: 0 # needed for git revparse and tag generation
# ------------------------------------------------------------------
# 2. Set up Docker Buildx (optional, but recommended for multiarch)
# ------------------------------------------------------------------
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
# ------------------------------------------------------------------
# 3. Log in to the Gitea container registry
# ------------------------------------------------------------------
- name: Log in to Gitea registry
uses: docker/login-action@v3
with:
registry: ${{ secrets.REGISTRY_HOST }} # e.g. registry.example.com
username: ${{ secrets.REGISTRY_USER }} # e.g. admin
password: ${{ secrets.REGISTRY_PASSWORD }} # e.g. <apitoken>
- name: Cache Docker layers
uses: actions/cache@v3
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-
# ------------------------------------------------------------------
# 4. Build the Docker image
# ------------------------------------------------------------------
- name: Build image
id: build
uses: docker/build-push-action@v5
with:
context: .
file: TinfoilVibeServer/Dockerfile
# do not push yet
push: false
tags: |
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ github.sha }}
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ steps.docker_tag_from_ref.outputs.docker-tag }}
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:latest
build-args: |
# Add any build args here
# ARG_NAME=VALUE
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache
# ------------------------------------------------------------------
# 5. Push the image to the registry
# ------------------------------------------------------------------
- name: Push image
uses: docker/build-push-action@v5
with:
context: .
file: TinfoilVibeServer/Dockerfile
push: true
tags: |
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ github.sha }}
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ steps.docker_tag_from_ref.outputs.docker-tag }}
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:latest
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache
# ------------------------------------------------------------------
# 6. (Optional) Clean up local Docker cache
# ------------------------------------------------------------------
- name: Docker system prune
run: docker system prune -f
if: ${{ always() }}
# ------------------------------------------------------------------
# 7. Output useful info
# ------------------------------------------------------------------
- name: Show pushed image tags
run: |
echo "Pushed image tags:"
echo "- ${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ github.sha }}"
echo "- ${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ steps.docker_tag_from_ref.outputs.docker-tag }}"
echo "- ${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:latest"
+74
View File
@@ -0,0 +1,74 @@
name: ci
on: [push, pull_request]
jobs:
build_linux:
runs-on: ubuntu-latest
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Convert to lowercase
id: github_repository_to_lowercase
run: |
# Grab the value (you can also use `${{ github.ref }}`, `${{ secrets.MY_SECRET }}`, etc.)
raw_value="${{ github.repository }}"
# Convert to lower case
lower_value=$(echo "$raw_value" | tr '[:upper:]' '[:lower:]')
# Export it to the workflow environment
# echo "MY_LOWER=$lower_value" >> $GITHUB_ENV
# If you want to use it as an output of this step:
echo "lowercase=$lower_value" >> $GITHUB_OUTPUT
- name: Convert ref to buildx safe value
id: docker_tag_from_ref
shell: bash
run: |
# Grab the raw ref
REF="${{ github.ref }}"
# Strip the "refs/*/" prefix (refs/heads/, refs/tags/…)
TAG=${REF#refs/*/}
# Replace characters that Docker tags disallow
# * "/" → "-"
# * ":" → "-"
# * Any other nonalphanumeric / . / _ / - → "-"
TAG=${TAG//\//-}
TAG=${TAG//:/-}
TAG=${TAG//[^a-zA-Z0-9._-]/-}
# (Optional) force lowercase Docker tags are casesensitive,
# but many people prefer lowercase
TAG=${TAG,,}
# Export to the action's output
echo "docker-tag=${TAG}" >> $GITHUB_OUTPUT
- name: Cache Docker layers
uses: actions/cache@v3
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-
- name: Build image
id: build
uses: docker/build-push-action@v5
with:
context: .
file: TinfoilVibeServer/Dockerfile
# do not push yet
push: false
tags: |
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ github.sha }}
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:${{ steps.docker_tag_from_ref.outputs.docker-tag }}
${{ vars.REGISTRY_HOST }}/${{ steps.github_repository_to_lowercase.outputs.lowercase }}:latest
build-args: |
# Add any build args here
# ARG_NAME=VALUE
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache
+8 -1
View File
@@ -2,4 +2,11 @@ bin/
obj/
/packages/
riderModule.iml
/_ReSharper.Caches/
/_ReSharper.Caches/
TinfoilVibeServer.sln.DotSettings.user
data/*
!.data/.gitkeep
**/*.local.json
TinfoilVibeServer/config/prod.keys
TinfoilVibeServer/data/*
!TinfoilVibeServer/data/.gitkeep
+18
View File
@@ -0,0 +1,18 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="TinfoilVibeServer: http" type="LaunchSettings" factoryName=".NET Launch Settings Profile">
<option name="LAUNCH_PROFILE_PROJECT_FILE_PATH" value="$PROJECT_DIR$/TinfoilVibeServer/TinfoilVibeServer.csproj" />
<option name="LAUNCH_PROFILE_TFM" value="net9.0" />
<option name="LAUNCH_PROFILE_NAME" value="http" />
<option name="USE_EXTERNAL_CONSOLE" value="0" />
<option name="USE_MONO" value="0" />
<option name="RUNTIME_ARGUMENTS" value="" />
<option name="GENERATE_APPLICATIONHOST_CONFIG" value="1" />
<option name="SHOW_IIS_EXPRESS_OUTPUT" value="0" />
<option name="SEND_DEBUG_REQUEST" value="1" />
<option name="ADDITIONAL_IIS_EXPRESS_ARGUMENTS" value="" />
<option name="AUTO_ATTACH_CHILDREN" value="0" />
<method v="2">
<option name="Build" />
</method>
</configuration>
</component>
+54
View File
@@ -0,0 +1,54 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="TinfoilVibeServer/Dockerfile" type="docker-deploy" factoryName="dockerfile" server-name="Docker WSL">
<deployment type="dockerfile">
<settings>
<option name="imageTag" value="gitea.ecenshu.net/ecenshu/tinfoilvibeserver:dev" />
<option name="containerName" value="tinfoilvibeserver" />
<option name="contextFolderPath" value="." />
<option name="envVars">
<list>
<DockerEnvVarImpl>
<option name="name" value="uid" />
<option name="value" value="1034" />
</DockerEnvVarImpl>
</list>
</option>
<option name="portBindings">
<list>
<DockerPortBindingImpl>
<option name="containerPort" value="8080" />
<option name="hostIp" value="127.0.0.1" />
<option name="hostPort" value="8080" />
</DockerPortBindingImpl>
</list>
</option>
<option name="showCommandPreview" value="true" />
<option name="sourceFilePath" value="TinfoilVibeServer/Dockerfile" />
<option name="volumeBindings">
<list>
<DockerVolumeBindingImpl>
<option name="containerPath" value="/app/data" />
<option name="hostPath" value="D:\Cloud\Git\TinfoilVibeServer\TinfoilVibeServer\bin\Debug\net9.0\data" />
</DockerVolumeBindingImpl>
<DockerVolumeBindingImpl>
<option name="containerPath" value="/app/config" />
<option name="hostPath" value="D:\Cloud\Git\TinfoilVibeServer\TinfoilVibeServer\bin\Debug\net9.0\config" />
</DockerVolumeBindingImpl>
<DockerVolumeBindingImpl>
<option name="containerPath" value="/roms_cold" />
<option name="hostPath" value="Z:\downloads\roms\switch" />
<option name="readOnly" value="true" />
</DockerVolumeBindingImpl>
<DockerVolumeBindingImpl>
<option name="containerPath" value="/roms_hot" />
<option name="hostPath" value="Z:\imgs\roms\Switch" />
<option name="readOnly" value="true" />
</DockerVolumeBindingImpl>
</list>
</option>
</settings>
</deployment>
<EXTENSION ID="com.jetbrains.rider.docker.debug" isFastModeEnabled="true" isSslEnabled="false" />
<method v="2" />
</configuration>
</component>
+1
View File
@@ -3,6 +3,7 @@ Microsoft Visual Studio Solution File, Format Version 12.00
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{41999D26-405C-4DAB-8991-CBA992117C84}"
ProjectSection(SolutionItems) = preProject
compose.yaml = compose.yaml
compose.overide.yaml = compose.overide.yaml
EndProjectSection
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TinfoilVibeServer", "TinfoilVibeServer\TinfoilVibeServer.csproj", "{DE992FDB-6D13-4152-925D-29D39A23FB75}"
+3
View File
@@ -87,7 +87,10 @@
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUniqueRef_00601_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FDecompilerCache_003Fdecompiler_003F77078e9a1d254191bb508f54a277fc6e1c2e00_003Fa0_003F631946a0_003FUniqueRef_00601_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUnpack_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FSourcesCache_003F4c375bcb6a2d2378855cad1b1c7cfe7ca1448866f1e8af44226775b5f75df86_003FUnpack_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUnsafeHelpers_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FDecompilerCache_003Fdecompiler_003F77078e9a1d254191bb508f54a277fc6e1c2e00_003Fd1_003Fc59f91c2_003FUnsafeHelpers_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUriExt_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E3_003Fresharper_002Dhost_003FSourcesCache_003F25edf3e734d0c76e87b29a9954b8b4a7383648a69396554742e5529205e2dd7_003FUriExt_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUriSyntax_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E3_003Fresharper_002Dhost_003FSourcesCache_003F7ed82aa0f48a6bf284b4aba7c70aff35142349e44fb4f6caec3d71611f9929_003FUriSyntax_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AUtility_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FSourcesCache_003F3ea3ed6216d2412ac7c33016ad940618bcfbcafe1633dc26832be514633b4_003FUtility_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AWaitHandle_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E3_003Fresharper_002Dhost_003FSourcesCache_003Fcab54c7c1f8d9b3da5f7878e39faba52467f16d0899a2c4b10086cb2ef73f2b_003FWaitHandle_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AXciPartition_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FDecompilerCache_003Fdecompiler_003F77078e9a1d254191bb508f54a277fc6e1c2e00_003F04_003F4e8815da_003FXciPartition_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
<s:String x:Key="/Default/CodeInspection/ExcludedFiles/FilesAndFoldersToSkip2/=7020124F_002D9FFC_002D4AC3_002D8F3D_002DAAB8E0240759_002Ff_003AZipArchive_002Ecs_002Fl_003AC_0021_003FUsers_003Fecens_003FAppData_003FRoaming_003FJetBrains_003FRider2025_002E2_003Fresharper_002Dhost_003FSourcesCache_003F39ececcf144e1f9c884152723ed93931cd232485eaf2824bf5beb526f1f321b_003FZipArchive_002Ecs/@EntryIndexedValue">ForceIncluded</s:String>
+90 -55
View File
@@ -12,7 +12,7 @@ public interface IAuthStore
void Dispose();
bool TryValidate(string username,
string password,
int? uid,
string? uid,
string ip,
out string? error);
@@ -29,33 +29,41 @@ public class AuthStore : IDisposable, IAuthStore
{
private readonly ILogger<AuthStore> _logger;
private readonly ConfigManager _configManager;
private readonly IHostEnvironment _env;
public readonly ConcurrentDictionary<string, Credential> Credentials = new();
public readonly ConcurrentDictionary<string, List<int>> Fingerprints = new();
public readonly ConcurrentDictionary<string, int> FailedAttempts = new();
private readonly HashSet<string> BlacklistIPs = new();
public readonly ConcurrentDictionary<string, Credential> Credentials = new();
public readonly ConcurrentDictionary<string, List<string>> Fingerprints = new();
public readonly ConcurrentDictionary<string, int> FailedAttempts = new();
private readonly HashSet<string> BlacklistIPs = new();
private readonly object _sync = new();
private readonly FileSystemWatcher _credentialsWatcher;
public AuthStore(ILogger<AuthStore> logger, ConfigManager configManager)
public AuthStore(ILogger<AuthStore> logger, ConfigManager configManager, IHostEnvironment env)
{
_logger = logger;
_configManager = configManager;
LoadAll();
_env = env;
var directoryName = Path.GetDirectoryName(_configManager.Settings.CredentialsFile);
LoadAll();
var credentialsFilePath = DetermineCredentialsPath(_configManager.Settings?.CredentialsFile, env);
var directoryName = Path.GetDirectoryName(Path.Combine(credentialsFilePath));
_credentialsWatcher = new FileSystemWatcher
{
Path = (!string.IsNullOrEmpty(directoryName))?directoryName : AppContext.BaseDirectory,
Filter = Path.GetFileName(_configManager.Settings.CredentialsFile),
Path = (!string.IsNullOrEmpty(directoryName)) ? directoryName : AppContext.BaseDirectory,
Filter = Path.GetFileName(credentialsFilePath) ?? throw new InvalidOperationException(),
NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.Size | NotifyFilters.Attributes
};
_credentialsWatcher.Changed += (_, _) => OnCredentialsChanged();
_credentialsWatcher.EnableRaisingEvents = true;
}
private static string DetermineCredentialsPath(string? settingsCredentialsFile, IHostEnvironment env)
{
if (settingsCredentialsFile == null) return Path.Combine("app","data","credentials.json");
return Path.IsPathRooted(settingsCredentialsFile) ? settingsCredentialsFile : Path.Combine(env.ContentRootPath,"data",settingsCredentialsFile);
}
public void Dispose()
{
_credentialsWatcher?.Dispose();
@@ -65,25 +73,26 @@ public class AuthStore : IDisposable, IAuthStore
private void LoadAll()
{
_logger.LogInformation("Loading authentication data from {File}", _configManager.Settings.CredentialsFile);
var credentialsFilePath = DetermineCredentialsPath(_configManager.Settings?.CredentialsFile, _env);
_logger.LogInformation("Loading authentication data from {File}", credentialsFilePath);
// credentials
if (File.Exists(_configManager.Settings.CredentialsFile))
if (File.Exists(credentialsFilePath))
{
var txt = File.ReadAllText(_configManager.Settings.CredentialsFile);
var txt = File.ReadAllText(credentialsFilePath);
var dict = JsonSerializer.Deserialize<Dictionary<string, Credential>>(txt)!;
foreach (var kv in dict)
Credentials[kv.Key] = kv.Value;
}
else
{
FileSystemExtensions.EnsureDirectoryExists(Path.GetDirectoryName(Path.GetFullPath(_configManager.Settings.CredentialsFile)));
FileSystemExtensions.EnsureDirectoryExists(Path.GetDirectoryName(Path.GetFullPath(_configManager.Settings?.CredentialsFile ?? throw new InvalidOperationException())));
}
// fingerprints
if (File.Exists(_configManager.Settings.FingerprintsFile))
{
var txt = File.ReadAllText(_configManager.Settings.FingerprintsFile);
var dict = JsonSerializer.Deserialize<Dictionary<string, List<int>>>(txt)!;
var dict = JsonSerializer.Deserialize<Dictionary<string, List<string>>>(txt)!;
foreach (var kv in dict)
Fingerprints[kv.Key] = kv.Value;
}
@@ -104,6 +113,7 @@ public class AuthStore : IDisposable, IAuthStore
{
FileSystemExtensions.EnsureDirectoryExists(Path.GetDirectoryName(Path.GetFullPath(_configManager.Settings.BlacklistFile)));
}
_logger.LogInformation("Loaded {UserCount} users, {FpCount} fingerprints, {IpCount} IPs",
Credentials.Count, Fingerprints.Count, BlacklistIPs.Count);
}
@@ -124,15 +134,17 @@ public class AuthStore : IDisposable, IAuthStore
private void ReloadCredentials()
{
if (!File.Exists(_configManager.Settings.CredentialsFile))
var credentialsFilePath = DetermineCredentialsPath(_configManager.Settings?.CredentialsFile, _env);
if (!File.Exists(credentialsFilePath))
{
_logger.LogError("Credentials file {File} does not exist", _configManager.Settings.CredentialsFile);
_logger.LogError("Credentials file {File} does not exist", credentialsFilePath);
return;
}
try
{
var txt = File.ReadAllText(_configManager.Settings.CredentialsFile);
var txt = File.ReadAllText(credentialsFilePath);
var newDict = JsonSerializer.Deserialize<Dictionary<string, Credential>>(txt)!;
lock (_sync)
@@ -142,9 +154,9 @@ public class AuthStore : IDisposable, IAuthStore
{
if (Credentials.TryGetValue(kv.Key, out var existing))
{
existing.PasswordHash = kv.Value.PasswordHash;
existing.PasswordHash = kv.Value.PasswordHash;
existing.AllowedUidCount = kv.Value.AllowedUidCount;
existing.Verified = kv.Value.Verified;
existing.Verified = kv.Value.Verified;
}
else
{
@@ -162,9 +174,9 @@ public class AuthStore : IDisposable, IAuthStore
}
}
}
catch(Exception ex)
catch (Exception ex)
{
_logger.LogError(ex, "Failed to reload credentials from {File}", _configManager.Settings.CredentialsFile);
_logger.LogError(ex, "Failed to reload credentials from {File}", credentialsFilePath);
// ignore malformed JSON or IO error keep old state
}
}
@@ -173,11 +185,7 @@ public class AuthStore : IDisposable, IAuthStore
#region Authentication logic
public bool TryValidate(string username,
string password,
int? uid,
string ip,
out string? error)
public bool TryValidate(string username, string password, string? uid, string ip, out string? error)
{
error = null;
lock (_sync)
@@ -190,14 +198,6 @@ public class AuthStore : IDisposable, IAuthStore
PersistCredentials();
_logger.LogInformation("Created new user {Username} (verified={Verified})", username, cred.Verified);
var list = Fingerprints.GetOrAdd(username, _ => new List<int>());
if (uid.HasValue && !list.Contains(uid.Value))
{
if (list.Count < cred.AllowedUidCount)
list.Add(uid.Value);
PersistFingerprints();
}
error = "User not verified";
IncrementFailed(username, ip);
return false;
@@ -218,15 +218,15 @@ public class AuthStore : IDisposable, IAuthStore
return false;
}
if (uid.HasValue)
if (uid != null)
{
var list = Fingerprints.GetOrAdd(username, _ => new List<int>());
var list = Fingerprints.GetOrAdd(username, _ => new List<string>());
if (!list.Contains(uid.Value))
if (!list.Contains(uid))
{
if (list.Count < cred.AllowedUidCount)
{
list.Add(uid.Value);
list.Add(uid);
PersistFingerprints();
}
else
@@ -237,6 +237,11 @@ public class AuthStore : IDisposable, IAuthStore
}
}
}
else
{
_logger.LogWarning("No Fingerprint given during authentication for {Username} from {IP}", username, ip);
return false;
}
FailedAttempts[username] = 0;
return true;
@@ -245,23 +250,30 @@ public class AuthStore : IDisposable, IAuthStore
public int IncrementFailed(string username, string ip)
{
var newCount = FailedAttempts.GetOrAdd(username, 0) + 1;
lock (_sync)
{
var newCount = FailedAttempts.GetOrAdd(username, 0) + 1;
FailedAttempts[username] = newCount;
}
_logger.LogInformation("Failed attempts for {Username} increased to {Count}", username, newCount);
_logger.LogInformation("Failed attempts for {Username} increased to {Count}", username, newCount);
if (newCount < _configManager.Settings.MaxFailedAttempts+1) return newCount;
BlacklistIPs.Add(ip);
PersistBlacklist();
lock (_sync)
{
FailedAttempts[username] = 0;
if (_configManager.Settings == null)
{
_logger.LogCritical("Settings not set to determine failed login counts");
return int.MaxValue;
}
if (newCount < _configManager.Settings.MaxFailedAttempts + 1) return newCount;
BlacklistIPs.Add(ip);
PersistBlacklist();
lock (_sync)
{
FailedAttempts[username] = 0;
}
_logger.LogWarning("IP {IP} blacklisted after {Count} failures", ip, newCount);
return newCount;
}
_logger.LogWarning("IP {IP} blacklisted after {Count} failures", ip, newCount);
return newCount;
}
#endregion
@@ -280,25 +292,48 @@ public class AuthStore : IDisposable, IAuthStore
private void PersistCredentials()
{
var credentialsFilePath = DetermineCredentialsPath(_configManager.Settings?.CredentialsFile, _env);
var json = JsonSerializer.Serialize(Credentials, new JsonSerializerOptions { WriteIndented = true });
File.WriteAllText(_configManager.Settings.CredentialsFile, json);
if (_configManager.Settings == null)
{
_logger.LogCritical("Credentials file not set, cannot persist");
}
else
{
File.WriteAllText(credentialsFilePath, json);
}
}
private void PersistFingerprints()
{
var json = JsonSerializer.Serialize(Fingerprints, new JsonSerializerOptions { WriteIndented = true });
File.WriteAllText(_configManager.Settings.FingerprintsFile, json);
if (_configManager.Settings == null)
{
_logger.LogCritical("Fingerprint file not set, cannot persist");
}
else
{
File.WriteAllText(_configManager.Settings.FingerprintsFile, json);
}
}
private void PersistBlacklist()
{
var json = JsonSerializer.Serialize(BlacklistIPs.ToArray(), new JsonSerializerOptions { WriteIndented = true });
File.WriteAllText(_configManager.Settings.BlacklistFile, json);
if (_configManager.Settings == null)
{
_logger.LogCritical("Blacklist file not set, cannot persist");
}
else
{
File.WriteAllText(_configManager.Settings.BlacklistFile, json);
}
}
#endregion
#region Blacklist helpers
public bool IsIPBlacklisted(string ipAddress)
{
return BlacklistIPs.Contains(ipAddress);
@@ -16,7 +16,7 @@ public class CancelableFileResult : FileResult
/// Allows you to set a suggested download name.
/// It will be sent in a “ContentDisposition” header.
/// </summary>
public string? FileDownloadName { get; set; }
public new string? FileDownloadName { get; set; }
public override async Task ExecuteResultAsync(ActionContext context)
{
@@ -94,7 +94,7 @@ public sealed class IndexController : ControllerBase
FileMode.Open,
FileAccess.Read,
FileShare.Read,
bufferSize: 128 * 1024 * 1024, // 81920, // 80KiB
bufferSize: 128 * 1024 * 1024, // 81920, // 80 KiB
useAsync: true); // <‑‑ VERY important for scalability
// 2️⃣ Return a cancellationaware result
@@ -42,7 +42,7 @@ public sealed class BasicAuthMiddleware
{
logger.LogWarning("Missing Authorization header from {IP}", ip);
Challenge(context);
logger.LogInformation("Sent 401 challenge to client");
logger.LogTrace("Sent 401 challenge to client");
return;
}
@@ -50,14 +50,14 @@ public sealed class BasicAuthMiddleware
if (!authHeader.StartsWith("Basic ", StringComparison.OrdinalIgnoreCase))
{
Challenge(context);
logger.LogInformation("Sent 401 challenge to client");
logger.LogTrace("Sent 401 challenge to client");
return;
}
string decoded;
try
{
var b64 = authHeader[6..].Trim();
var b64 = authHeader.Substring(6).Trim();
decoded = Encoding.UTF8.GetString(Convert.FromBase64String(b64));
}
catch
@@ -79,12 +79,9 @@ public sealed class BasicAuthMiddleware
var password = parts[1];
// 3) UID header (optional)
int? uid = null;
string? uid = null;
if (context.Request.Headers.TryGetValue("UID", out var uidHeader))
{
if (int.TryParse(uidHeader.ToString(), out var parsedUid))
uid = parsedUid;
}
uid = uidHeader.FirstOrDefault();
// 4) Validate
if (!store.TryValidate(username, password, uid, ip, out var error))
+1 -2
View File
@@ -10,6 +10,5 @@ public sealed record AppSettings(
string CredentialsFile,
string FingerprintsFile,
string BlacklistFile,
int MaxFailedAttempts,
string KeySetFile
int MaxFailedAttempts
);
+1 -1
View File
@@ -8,6 +8,6 @@ namespace TinfoilVibeServer.Models;
public sealed record FileEntry(
string Path, // nsp or archive path
long Size, // size of nsp or full archive
string Hash, // SHA256 hex of first NCA of first NCP in NSP or archive
string? Hash, // SHA256 hex of first NCA of first NCP in NSP or archive
List<NcaMetadataWithHash> Titles // Details of all NSP Roms in the Path
);
@@ -2,5 +2,9 @@
public class IndexBuilderSettings
{
public string CacheFilePath { get; set; } = "indexcache.json";
public string ApiBaseUrl { get; set; } = "http://tinfoil.localhost";
public ICollection<string>? IndexDirectories { get; set; }
public string? LoginMessage { get; set; }
}
@@ -0,0 +1,26 @@
using LibHac.Ncm;
namespace TinfoilVibeServer.Models;
/// <summary>
/// DTO returned by the extractor contains all data the snapshot needs.
/// </summary>
public sealed class NcaMetadataWithHash
{
public string TitleId { get; }
public string ApplicationTitle { get; set; }
public int Version { get; }
public ContentMetaType ContentMetaType { get; set; }
public string? Hash { get; }
public NcaMetadataWithHash(string titleId, string applicationTitle, int version,
ContentMetaType contentMetaType, string? hash = null)
{
TitleId = titleId;
ApplicationTitle = applicationTitle;
Version = version;
ContentMetaType = contentMetaType;
Hash = hash;
}
}
+24 -15
View File
@@ -1,36 +1,41 @@
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
namespace TinfoilVibeServer.Models;
public sealed class SnapshotOptions : INotifyPropertyChanged
{
private List<string> _rootDirectories = new();
private List<string> _rootDirectories = [];
public List<string> RootDirectories
{
get => _rootDirectories;
set
{
if (_rootDirectories != value)
{
_rootDirectories = value;
OnPropertyChanged(nameof(RootDirectories));
}
if (_rootDirectories.Except(value) == Array.Empty<string>()) return;
_rootDirectories = value;
OnPropertyChanged(nameof(RootDirectories));
}
}
private List<string> _archiveExtensions = new();
public List<string> ArchiveExtensions
{
get => _archiveExtensions;
set
{
if (_archiveExtensions != value)
{
_archiveExtensions = value;
OnPropertyChanged(nameof(_archiveExtensions));
}
if (_archiveExtensions.Except(value) == Array.Empty<string>()) return;
_archiveExtensions = value;
OnPropertyChanged(nameof(_archiveExtensions));
}
}
private List<string> _romExtensions = new();
[Required(ErrorMessage = "No ROM extension specified")]
public List<string> RomExtensions
{
get => _romExtensions;
@@ -43,7 +48,9 @@ public sealed class SnapshotOptions : INotifyPropertyChanged
}
}
}
private TimeSpan _cacheTtl = TimeSpan.FromHours(1);
public TimeSpan CacheTtl
{
get => _cacheTtl;
@@ -56,7 +63,7 @@ public sealed class SnapshotOptions : INotifyPropertyChanged
}
}
}
private string _snapshotFile = "snapshot.json";
public string SnapshotFile
@@ -64,25 +71,27 @@ public sealed class SnapshotOptions : INotifyPropertyChanged
get => _snapshotFile;
set
{
if (string.Equals(_snapshotFile,value, StringComparison.InvariantCultureIgnoreCase)) return;
if (string.Equals(_snapshotFile, value, StringComparison.InvariantCultureIgnoreCase)) return;
_snapshotFile = value;
OnPropertyChanged(nameof(SnapshotFile));
}
}
private string _snapshotBackupFile = "snapshot.bak";
public string SnapshotBackupFile
{
get => _snapshotBackupFile;
set
{
if (string.Equals(_snapshotBackupFile,value, StringComparison.InvariantCultureIgnoreCase)) return;
if (string.Equals(_snapshotBackupFile, value, StringComparison.InvariantCultureIgnoreCase)) return;
_snapshotBackupFile = value;
OnPropertyChanged(nameof(SnapshotBackupFile));
}
}
public event PropertyChangedEventHandler? PropertyChanged;
private void OnPropertyChanged(string propertyName) =>
PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(propertyName));
}
+45 -16
View File
@@ -1,3 +1,4 @@
using System.Text.Json;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Options;
using TinfoilVibeServer.Authentication;
@@ -9,31 +10,59 @@ var builder = WebApplication.CreateBuilder(args);
builder.Logging.ClearProviders();
builder.Logging.AddConsole();
builder.Logging.AddDebug();
// -------------------------------------------------------------------
// 1) Configuration read appsettings.json once and expose it via
// ConfigManager (reloads on file change)
// -------------------------------------------------------------------
builder.Services.AddMemoryCache();
builder.Services.Configure<TitleDbOptions>(builder.Configuration.GetSection("TitleDb"));
builder.Services.Configure<AuthSettings>(builder.Configuration.GetSection("AuthSettings"));
builder.Services.Configure<SnapshotOptions>(builder.Configuration.GetSection("Snapshot"));
builder.Services.AddSingleton<ConfigManager>();
builder.Services.AddSingleton<INSPExtractor, NSPExtractor>(sp =>
var dataRoot = builder.Configuration["CONFIG_ROOT"] ?? "/app/config/";
// 1️⃣ Load the embedded default
var defaultResource = typeof(Program).Assembly
.GetManifestResourceStream("TinfoilVibeServer.appsettings.default.json")!; // adjust namespace
var defaultConfig = JsonDocument.Parse(defaultResource).RootElement;
// 2️⃣ Try to write the file if it doesn't exist
var configPath = Path.Combine(dataRoot, "appsettings.json");
if (!File.Exists(configPath))
{
var config = sp.GetRequiredService<ConfigManager>();
var logger = sp.GetRequiredService<ILogger<INSPExtractor>>();
var keySet = KeySetHolder.KeySet; // already loaded by ConfigManager
return new NSPExtractor(keySet, logger);
});
// write the embedded JSON straight to disk
try
{
File.WriteAllText(configPath, defaultConfig.GetRawText());
}
catch (Exception e)
{
var tempFactory = LoggerFactory.Create(loggingBuilder =>
{
loggingBuilder.AddConsole();
loggingBuilder.AddDebug();
});
var logger = tempFactory.CreateLogger<Program>();
logger.LogError(e, "Failed to write default config file");
}
}
var config = new ConfigurationBuilder()
.AddJsonFile(Path.Combine(dataRoot,"appsettings.json"), optional: false, reloadOnChange: true)
.AddJsonFile(Path.Combine(dataRoot,$"appsettings.{builder.Environment.EnvironmentName}.json"), optional: true, reloadOnChange: true)
.AddJsonFile(Path.Combine(dataRoot,"appsettings.local.json"), optional: true, reloadOnChange: true)
.AddEnvironmentVariables()
.AddCommandLine(args).Build();
builder.Configuration.AddConfiguration(config);
builder.Services.AddOptions();
builder.Services.Configure<TitleDbOptions>(builder.Configuration.GetSection("TitleDb"));
builder.Services.Configure<NSPExtractorOptions>(builder.Configuration.GetSection("NSPExtractor"));
builder.Services.Configure<AuthSettings>(builder.Configuration.GetSection("AuthSettings"));
builder.Services.Configure<IndexBuilderSettings>(builder.Configuration.GetSection("IndexBuilder"));
builder.Services.AddOptions<SnapshotOptions>().Bind(builder.Configuration.GetSection("Snapshot")).ValidateOnStart();
builder.Services.AddSingleton<ConfigManager>();
builder.Services.AddSingleton<INSPExtractor, NSPExtractor>();
builder.Services.AddSingleton<SnapshotService>();
builder.Services.AddSingleton<ISnapshotService, SnapshotService>(sp => sp.GetRequiredService<SnapshotService>());
builder.Services.AddSingleton<IAuthStore, AuthStore>();
builder.Services.AddSingleton<TitleDatabaseService>();
builder.Services.AddSingleton<IArchiveHandler, ArchiveHandler>();
builder.Services.AddSingleton<IndexBuilderService>();
builder.Services.AddHostedService<IndexBuilderService>(provider => provider.GetRequiredService<IndexBuilderService>());
builder.Services.AddHostedService<SnapshotService>(provider => provider.GetRequiredService<SnapshotService>());
builder.Services.AddHostedService<TitleDatabaseService>(provider => provider.GetRequiredService<TitleDatabaseService>()).AddHttpClient();
builder.Services.AddHostedService<IndexBuilderService>(provider => provider.GetRequiredService<IndexBuilderService>());
builder.Services.AddControllers(); // add MVC
// -------------------------------------------------------------------
// 2) Middleware BasicAuth (verifies username, password, UID, blacklist)
@@ -54,7 +83,7 @@ app.MapGet("/debug", () => new SnapshotService(
app.Services.GetRequiredService<IOptionsMonitor<SnapshotOptions>>(),
app.Services.GetRequiredService<INSPExtractor>(),
app.Services.GetRequiredService<IArchiveHandler>(),
app.Services.GetRequiredService<ILogger<SnapshotService>>())
app.Services.GetRequiredService<ILogger<SnapshotService>>(), app.Services.GetRequiredService<IHostEnvironment>())
.GetSnapshot());
app.Lifetime.ApplicationStarted.Register(() =>
app.Services.GetRequiredService<ILogger<Program>>().LogInformation("Application started. Listening on {Urls}", string.Join(", ", app.Urls)));
@@ -5,9 +5,10 @@
"commandName": "Project",
"dotnetRunMessages": true,
"launchBrowser": false,
"applicationUrl": "http://192.168.1.145:80;http://tinfoil.localhost:8080;http://tinfoil.ecenshu.net",
"applicationUrl": "http://192.168.1.145:8080;http://tinfoil.localhost:8081;http://tinfoil.ecenshu.net",
"environmentVariables": {
"ASPNETCORE_ENVIRONMENT": "Development"
"ASPNETCORE_ENVIRONMENT": "Development",
"CONFIG_ROOT": "./config/"
}
},
"https": {
+1 -1
View File
@@ -97,7 +97,7 @@ public sealed class ArchiveHandler : IArchiveHandler
using var archive = SevenZipArchive.Open(path);
foreach (var entry in archive.Entries)
{
if (!entry.IsDirectory && IsRomArchive(entry.Key))
if (!entry.IsDirectory && entry.Key != null && IsRomArchive(entry.Key))
{
var temp = Path.GetTempFileName();
entry.WriteToFile(temp);
+10 -34
View File
@@ -1,5 +1,4 @@
using System.IO;
using System.Text.Json;
using System.Text.Json;
using LibHac.Common.Keys;
using TinfoilVibeServer.Models;
@@ -11,17 +10,16 @@ namespace TinfoilVibeServer.Services;
/// </summary>
public class ConfigManager
{
public AppSettings Settings { get; private set; }
public event Action<AppSettings>? OnChange;
public AppSettings? Settings { get; private set; }
public event Action<AppSettings?>? OnChange;
private readonly string _configPath;
private readonly FileSystemWatcher _watcher;
private readonly object _sync = new();
private readonly Lock _sync = new();
public ConfigManager()
{
_configPath = Path.Combine(AppContext.BaseDirectory, "appsettings.json");
_configPath = Path.Combine(AppContext.BaseDirectory, "config", "appsettings.json");
Load();
_watcher = new FileSystemWatcher
@@ -39,41 +37,19 @@ public class ConfigManager
if (!File.Exists(_configPath))
{
Settings = new AppSettings(
RootDirectories: Array.Empty<string>(),
WhitelistExtensions: Array.Empty<string>(),
RomExtensions: Array.Empty<string>(),
RootDirectories: [],
WhitelistExtensions: [],
RomExtensions: [],
CredentialsFile: "credentials.json",
FingerprintsFile: "fingerprints.json",
BlacklistFile: "blacklist.json",
MaxFailedAttempts: 5,
KeySetFile: "keys.bin"
MaxFailedAttempts: 5
);
return;
}
var txt = File.ReadAllText(_configPath);
Settings = JsonSerializer.Deserialize<AppSettings>(txt, new JsonSerializerOptions { PropertyNameCaseInsensitive = true })!;
// --- Load the KeySet --------------------------------------------
if (!string.IsNullOrWhiteSpace(Settings.KeySetFile))
{
var keyFilePath = Path.Combine(AppContext.BaseDirectory, Settings.KeySetFile);
if (File.Exists(keyFilePath))
{
// LibHac provides a static helper to load a keyset file.
// If the file is not found or corrupt, we simply keep the
// default (empty) key set the app will throw later
// when a title requires a missing key.
try
{
KeySetHolder.KeySet = ExternalKeyReader.ReadKeyFile(keyFilePath);
}
catch
{
KeySetHolder.KeySet = new KeySet(); // fallback
}
}
}
}
private void Reload()
@@ -11,29 +11,35 @@ namespace TinfoilVibeServer.Services;
// *** NEW ***
public sealed class IndexBuilderService: IHostedService
{
private const string CacheFileName = "indexcache.json";
private readonly IOptions<IndexBuilderSettings> _options;
private readonly IOptionsMonitor<IndexBuilderSettings> _options;
private readonly ISnapshotService _snapshotService;
private readonly TitleDatabaseService _titleDb;
private readonly IConfiguration _configuration;
private readonly ILogger<IndexBuilderService> _logger;
private readonly string _cachePath;
private readonly IHostEnvironment _hostEnvironment;
private readonly SemaphoreSlim _lock = new(1, 1);
public IndexBuilderService(
IOptions<IndexBuilderSettings> options,
IOptionsMonitor<IndexBuilderSettings> options,
ISnapshotService snapshotService,
TitleDatabaseService titleDb,
IConfiguration configuration,
ILogger<IndexBuilderService> logger)
ILogger<IndexBuilderService> logger,
IHostEnvironment hostEnvironment)
{
_options = options;
_snapshotService = snapshotService;
_titleDb = titleDb;
_configuration = configuration;
_logger = logger;
_cachePath = Path.Combine(AppContext.BaseDirectory, CacheFileName);
_hostEnvironment = hostEnvironment;
}
private static string DetermineCachePath(IOptionsMonitor<IndexBuilderSettings> options, IHostEnvironment environment)
{
if (Path.IsPathRooted(options.CurrentValue.CacheFilePath))
{
return options.CurrentValue.CacheFilePath;
}
return Path.Combine(environment.ContentRootPath, "data", options.CurrentValue.CacheFilePath);
}
public IndexDto Build(HttpContext httpContext)
@@ -60,10 +66,9 @@ public sealed class IndexBuilderService: IHostedService
// 3️⃣ Build new index from snapshot entries
var files = ParseSnapshotFiles(snapshot, new Uri(httpContext.Request.Scheme + "://" + httpContext.Request.Host + httpContext.Request.PathBase));
var directories = _configuration.GetSection("Directories")
.Get<string[]>() ?? Array.Empty<string>();
var directories = _options.CurrentValue.IndexDirectories ?? Array.Empty<string>();
var success = _configuration["SuccessMessage"] ?? string.Empty;
var success = _options.CurrentValue.LoginMessage ?? string.Empty;
var index = new IndexDto(files.SelectMany(inner => inner).ToList(), directories.ToList(), success);
@@ -114,9 +119,11 @@ public sealed class IndexBuilderService: IHostedService
var fileName =Uri.EscapeDataString($"{name}[{titleId}][v{versionNumberParsed}][{patchOrApp}].nsp");
var url = $"{baseUri.ToString().TrimEnd('/')}/{fileName}";
if (Uri.IsWellFormedUriString(url, UriKind.Absolute))
var isWellFormed = Uri.TryCreate(url, UriKind.Absolute, out Uri? parsedUri);
if (isWellFormed && parsedUri != null)
{
fileDtos.Add(new FileDto(url, e.Size));
fileDtos.Add(new FileDto(parsedUri.AbsoluteUri, e.Size));
}
else
{
@@ -133,18 +140,28 @@ public sealed class IndexBuilderService: IHostedService
private IndexCache? LoadCache()
{
if (!File.Exists(_cachePath)) return null;
var cachePath = DetermineCachePath(_options, _hostEnvironment);
if (!File.Exists(cachePath)) return null;
_lock.Wait();
var json = File.ReadAllText(_cachePath);
var json = File.ReadAllText(cachePath);
_lock.Release();
_logger.LogInformation("Loaded index cache from {Path}", _cachePath);
_logger.LogInformation("Loaded index cache from {Path}", cachePath);
return JsonSerializer.Deserialize<IndexCache>(json);
}
private void PersistCache(string snapshotHash, IndexDto index)
{
var cachePath = DetermineCachePath(_options, _hostEnvironment);
var cache = new IndexCache(snapshotHash, index);
File.WriteAllText(_cachePath, JsonSerializer.Serialize(cache, new JsonSerializerOptions{WriteIndented=true}));
File.WriteAllText(cachePath, JsonSerializer.Serialize(cache, new JsonSerializerOptions{WriteIndented=true}));
}
public void InvalidateIndex(object? sender, EventArgs e)
{
var cachePath = DetermineCachePath(_options, _hostEnvironment);
if (!File.Exists(cachePath)) return;
File.Delete(cachePath);
_logger.LogInformation("Index cache cleared");
}
private static string ComputeSnapshotHash(IEnumerable<FileEntry> entries)
@@ -161,27 +178,17 @@ public sealed class IndexBuilderService: IHostedService
public Task StartAsync(CancellationToken cancellationToken)
{
var url = new Uri(_options.Value.ApiBaseUrl);
var url = new Uri(_options.CurrentValue.ApiBaseUrl);
var host = url.Host;
Build(new DefaultHttpContext {HttpContext = { Request = { Host = new HostString(host), Scheme = url.Scheme, Path = new PathString(url.AbsolutePath)}}});
_snapshotService.SnapshotRebuilt += InvalidateIndex;
return Task.CompletedTask;
}
public void InvalidateIndex(object? sender, EventArgs e)
{
if (!File.Exists(_cachePath)) return;
File.Delete(_cachePath);
_logger.LogInformation("Index cache cleared");
}
public Task StopAsync(CancellationToken cancellationToken)
{
_snapshotService.SnapshotRebuilt -= InvalidateIndex;
return Task.CompletedTask; // nothing special to do on shutdown
}
#endregion
}
+58 -29
View File
@@ -9,6 +9,9 @@ using LibHac.Common.Keys;
using LibHac.Ncm;
using LibHac.Tools.Fs;
using LibHac.Tools.Ncm;
using Microsoft.Extensions.Options;
using TinfoilVibeServer.Models;
using Path = System.IO.Path;
namespace TinfoilVibeServer.Services
{
@@ -32,13 +35,49 @@ namespace TinfoilVibeServer.Services
/// </summary>
public sealed class NSPExtractor : INSPExtractor
{
private readonly KeySet _keySet;
private readonly ILogger<INSPExtractor> _logger;
private KeySet? _keySet;
public NSPExtractor(KeySet keySet, ILogger<INSPExtractor> logger)
public KeySet? KeySet
{
_keySet = keySet;
get
{
if (_keySet != null) return _keySet;
if (_options.CurrentValue.KeyFile == null) return null;
var dataRoot = _environment.ContentRootPath ?? "/app/config";
if (Path.IsPathRooted(_options.CurrentValue.KeyFile))
{
_keySet = ExternalKeyReader.ReadKeyFile(_options.CurrentValue.KeyFile);
}
else
{
_keySet = ExternalKeyReader.ReadKeyFile(Path.Combine(dataRoot, "config", _options.CurrentValue.KeyFile));
}
return _keySet;
}
}
private readonly IOptionsMonitor<NSPExtractorOptions> _options;
private readonly ILogger<INSPExtractor> _logger;
private readonly IHostEnvironment _environment;
public NSPExtractor(IOptionsMonitor<NSPExtractorOptions> options, ILogger<INSPExtractor> logger, IHostEnvironment environment)
{
_options = options;
_options.OnChange(o =>
{
if (o.KeyFile == null)
{
_logger?.LogInformation("No KeySet specified, skipping key validation");
}
if (!File.Exists(o.KeyFile))
{
_logger?.LogWarning("KeySet file {KeyFile} does not exist", o.KeyFile);
}
});
_logger = logger;
_environment = environment;
}
/// <summary>
@@ -55,6 +94,8 @@ namespace TinfoilVibeServer.Services
/// </summary>
public NcaMetadataWithHash? ExtractFromStream(Stream stream)
{
if (KeySet == null) return null;
if (!stream.CanSeek) return null;
stream.Seek(0, SeekOrigin.Begin);
@@ -67,7 +108,7 @@ namespace TinfoilVibeServer.Services
if (IsXciFileSystem(stream))
{
var xci = new Xci(_keySet, storage);
var xci = new Xci(KeySet, storage);
List<DirectoryEntryEx> ncaEntries;
if (xci.HasPartition(XciPartitionType.Secure))
{
@@ -85,7 +126,7 @@ namespace TinfoilVibeServer.Services
using var ncaFile = fileRef.Release();
using var ncaFileStorage = new FileStorage(ncaFile);
var nca = new Nca(_keySet, ncaFileStorage);
var nca = new Nca(KeySet, ncaFileStorage);
if (hash == null)
{
// Hash the *first* NCA stream the stream we just opened
@@ -112,6 +153,8 @@ namespace TinfoilVibeServer.Services
private NcaMetadataWithHash? ExtractNSPFromStream(StreamStorage storage)
{
if (KeySet == null) return null;
List<DirectoryEntryEx> ncaEntries;
_logger.LogInformation("Processing as NSP");
var partition = new PartitionFileSystem();
@@ -129,7 +172,7 @@ namespace TinfoilVibeServer.Services
using var ncaFile = fileRef.Release();
using var ncaFileStorage = new FileStorage(ncaFile);
var nca = new Nca(_keySet, ncaFileStorage);
var nca = new Nca(KeySet, ncaFileStorage);
if (hash == null)
{
// Hash the *first* NCA stream the stream we just opened
@@ -198,6 +241,8 @@ namespace TinfoilVibeServer.Services
}
private bool IsXciFileSystem(Stream stream)
{
if (KeySet == null) return false;
try
{
if (!stream.CanSeek) return false;
@@ -206,7 +251,7 @@ namespace TinfoilVibeServer.Services
var storage = new StreamStorage(stream, true);
try
{
var xciBlock = new Xci(_keySet, storage);
var xciBlock = new Xci(KeySet, storage);
_logger.LogInformation("XCI found");
return xciBlock.HasPartition(XciPartitionType.Secure);
}
@@ -225,6 +270,8 @@ namespace TinfoilVibeServer.Services
public string ExtractHashFromStream(Stream nspStream)
{
if (KeySet == null) return string.Empty;
if (!IsPfs0FileSystem(nspStream))
return string.Empty;
@@ -249,7 +296,7 @@ namespace TinfoilVibeServer.Services
try
{
var nca = new Nca(_keySet, ncaFileStorage);
var nca = new Nca(KeySet, ncaFileStorage);
if (nca.Header.ContentType != NcaContentType.Meta)
continue; // only the meta NCA contains title metadata
@@ -271,26 +318,8 @@ namespace TinfoilVibeServer.Services
}
}
/// <summary>
/// DTO returned by the extractor contains all data the snapshot needs.
/// </summary>
public sealed class NcaMetadataWithHash
public class NSPExtractorOptions
{
public string TitleId { get; }
public string ApplicationTitle { get; set; }
public int Version { get; }
public ContentMetaType ContentMetaType { get; set; }
public string Hash { get; }
public NcaMetadataWithHash(string titleId, string applicationTitle, int version,
ContentMetaType contentMetaType, string hash)
{
TitleId = titleId;
ApplicationTitle = applicationTitle;
Version = version;
ContentMetaType = contentMetaType;
Hash = hash;
}
public string? KeyFile { get; set; }
}
}
@@ -83,7 +83,7 @@ namespace TinfoilVibeServer.Services
continue;
// SharpCompress gives us a stream that must be disposed by the caller
yield return new RomArchiveEntry(entry.Key, entry.OpenEntryStream());
if (entry.Key != null) yield return new RomArchiveEntry(entry.Key, entry.OpenEntryStream());
}
}
}
+155 -131
View File
@@ -3,13 +3,14 @@ using System.Security.Cryptography;
using System.Text.Json;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Options;
using Microsoft.Extensions.Primitives;
using TinfoilVibeServer.Models;
using TinfoilVibeServer.Utilities;
namespace TinfoilVibeServer.Services;
public interface ISnapshotService
{
event EventHandler SnapshotRebuilt; // raised after a rebuild
event EventHandler SnapshotRebuilt; // event raised after a rebuild
void RebuildSnapshot();
SnapshotService.ROMSnapshot GetSnapshot();
@@ -33,6 +34,7 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
private readonly INSPExtractor _nspExtractor;
private readonly IArchiveHandler _archiveHandler;
private readonly ILogger<SnapshotService> _logger;
private readonly IHostEnvironment _environment;
private readonly string _jsonPath;
private readonly string _snapshotPath;
private readonly ConcurrentDictionary<string, SnapshotEntry> _cache = new();
@@ -54,67 +56,68 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
IOptionsMonitor<SnapshotOptions> options,
INSPExtractor nspExtractor,
IArchiveHandler archiveHandler,
ILogger<SnapshotService> logger)
ILogger<SnapshotService> logger,
IHostEnvironment environment
)
{
_options = options.CurrentValue;
_debouncerCache = debouncerCache;
_nspExtractor = nspExtractor;
_archiveHandler = archiveHandler;
_logger = logger;
_jsonPath = Path.Combine(AppContext.BaseDirectory, _options.SnapshotFile);
_environment = environment;
_jsonPath = Path.Combine(Path.DirectorySeparatorChar.ToString(),"app","data", _options.SnapshotFile);
// Debounce timer for persisting snapshot
long debounceTime = 200;
var entryOptions = new MemoryCacheEntryOptions()
.SetSlidingExpiration(TimeSpan.FromSeconds(debounceTime)).RegisterPostEvictionCallback((key, value, reason,
state) =>
{
_logger.LogInformation("Should persist the snapshot {Key}, {Reason}", key, reason);
}); // <‑‑ sliding!
FileSystemExtensions.EnsureDirectoryExists(Path.GetFullPath(Path.GetDirectoryName(_jsonPath)));
FileSystemExtensions.EnsureDirectoryExists(Path.GetFullPath(Path.GetDirectoryName(_jsonPath) ?? throw new InvalidOperationException()));
if (!File.Exists(_jsonPath))
{
_snapshotFileSemaphore.Wait();
File.WriteAllText(_jsonPath, "[]");
_snapshotFileSemaphore.Release();
}
_snapshotPath = Path.Combine(AppContext.BaseDirectory, _options.SnapshotBackupFile);
FileSystemExtensions.EnsureDirectoryExists(Path.GetFullPath(Path.GetDirectoryName(_snapshotPath)));
// 1️⃣ Register for *property* changes
_options.PropertyChanged += (s, e) => OnOptionsChanged(e.PropertyName);
_snapshotPath = Path.Combine(Path.DirectorySeparatorChar.ToString(),"app","data", _options.SnapshotBackupFile);
FileSystemExtensions.EnsureDirectoryExists(Path.GetFullPath(Path.GetDirectoryName(_snapshotPath) ?? throw new InvalidOperationException()));
// 1️⃣ Register for *property* changes
options.OnChange((snapshotOptions, arg) =>
{
_options.RootDirectories = snapshotOptions.RootDirectories;
});
_options.PropertyChanged += (s, e) => OnOptionsChanged(e.PropertyName);
if (_options.RootDirectories.Count == 0)
{
_logger.LogInformation("No directories set to watch for ROMS/Archives");
}
foreach (var path in _options.RootDirectories)
{
AddWatchDirectory(path);
}
}
// --------- Private helpers ---------
private void OnOptionsChanged(string propertyName)
private void OnOptionsChanged(string? propertyName)
{
if (propertyName == nameof(SnapshotOptions.RootDirectories))
if (propertyName != nameof(SnapshotOptions.RootDirectories)) return;
_logger.LogInformation("Root directories changed, rebuilding snapshot");
var fileSystemWatchers = _watchers.Where(watcher => !_options.RootDirectories.Contains(watcher.Path));
var systemWatchers = fileSystemWatchers.ToList();
foreach (var watcher in systemWatchers)
{
var fileSystemWatchers = _watchers.Where(watcher => !_options.RootDirectories.Contains(watcher.Path));
foreach (var watcher in fileSystemWatchers)
{
watcher.EnableRaisingEvents = false;
watcher.Dispose();
_watchers.Remove(watcher);
}
var newWatchedDirectories = _options.RootDirectories.Where(newWatchedDirectory =>
!_watchers.Any(watcher =>
string.Equals(watcher.Path, newWatchedDirectory, StringComparison.OrdinalIgnoreCase)));
foreach (var newWatchedDirectory in newWatchedDirectories)
{
AddWatchDirectory(newWatchedDirectory);
}
BuildSnapshotAsync(); // rebuild everything
PersistSnapshotAsync();
RemoveWatchDirectory(watcher.Path);
}
var newWatchedDirectories = _options.RootDirectories.Where(newWatchedDirectory =>
!_watchers.Any(watcher =>
string.Equals(watcher.Path, newWatchedDirectory, StringComparison.OrdinalIgnoreCase)));
foreach (var newWatchedDirectory in newWatchedDirectories)
{
AddWatchDirectory(newWatchedDirectory);
}
BuildSnapshotAsync(); // rebuild everything
PersistSnapshotAsync();
}
@@ -154,8 +157,10 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
private void ThrottleSnapshotUpdate(FileSystemEventArgs fileSystemEventArgs)
{
SnapshotRebuilding?.Invoke(this, fileSystemEventArgs);
CancellationTokenSource cts = new();
using var cacheEntry = _debouncerCache.CreateEntry(fileSystemEventArgs.FullPath)
//.SetAbsoluteExpiration(TimeSpan.FromMilliseconds(DebounceMs))
.AddExpirationToken(new CancellationChangeToken(cts.Token))
.SetValue(fileSystemEventArgs)
.SetOptions(new MemoryCacheEntryOptions
{
@@ -166,7 +171,7 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
EvictionCallback =
(key, value, reason, state) =>
{
if (reason != EvictionReason.Expired) return;
if (!(reason == EvictionReason.Expired || reason == EvictionReason.TokenExpired)) return;
if (value is FileSystemEventArgs args)
{
@@ -184,10 +189,9 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
}
}
});
cacheEntry.AbsoluteExpirationRelativeToNow = TimeSpan.FromMilliseconds(DebounceMs);
cts.CancelAfter(TimeSpan.FromMilliseconds(DebounceMs));
_logger.LogDebug("File system event {EventType} on {Path} at {Time}", fileSystemEventArgs.ChangeType,
fileSystemEventArgs.FullPath, DateTime.Now.ToString("HH:mm:ss"));
fileSystemEventArgs.FullPath, DateTime.Now.ToString("HH:mm:ss.fff"));
}
private static bool IsFileLocked(string filePath)
{
@@ -252,60 +256,40 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
var index = LoadSnapshotIndex();
var latestModifiedUtcParallel = FileSystemExtensions.GetLatestModifiedUtcParallel(_options.RootDirectories);
var fileInfo = new FileInfo(_snapshotPath);
bool snapshotVerified = true;
bool snapshotVerified = fileInfo.Exists;
if (latestModifiedUtcParallel.HasValue && latestModifiedUtcParallel.Value < fileInfo.LastWriteTimeUtc)
{
if (index.Count != 0)
{
// directory may have been added with older roms, verify that the snapshot is still up to date
foreach (var dir in _options.RootDirectories)
{
// check first entry is in index
var entry = BuildSnapshot(dir).FirstOrDefault();
if (entry != null)
var firstEntry = BuildSnapshot(dir).FirstOrDefault();
if (firstEntry != null && !index.TryGetValue(firstEntry.Path, out _))
{
if (!index.TryGetValue(entry.Path, out var cached))
{
snapshotVerified = false;
_logger.LogInformation("Snapshot does not contain first entry in directory {Directory}", dir);
}
snapshotVerified = false;
_logger.LogInformation("Snapshot does not contain first entry in directory {Directory}", dir);
}
}
}
}
if (snapshotVerified)
if (!snapshotVerified)
{
_logger.LogInformation("Rebuilding snapshot (root dirs: {Count})", _options.RootDirectories.Count);
var entries = new List<FileEntry>();
foreach (var dir in _options.RootDirectories)
{
foreach (var entry in BuildSnapshot(dir))
{
_logger.LogInformation("Snapshot is up to date");
return Task.CompletedTask;
if (entry != null) entries.Add(entry);
}
}
else
{
_logger.LogInformation("Snapshot is up to date but index is empty");
}
}
_logger.LogInformation("Rebuilding snapshot (root dirs: {Count})", _options.RootDirectories.Count);
var entries = new List<FileEntry>();
var snapshotChanged = false;
foreach (var dir in _options.RootDirectories)
{
_ = Task.Run(() =>
{
_logger.LogInformation("Rebuilding directory {Directory}", dir);
var buildSnapshot = BuildSnapshot(dir);
var fileEntries = buildSnapshot.ToList();
snapshotChanged = snapshotChanged || fileEntries.Count != 0;
entries.AddRange(fileEntries.Where(entry => entry != null)!);
});
var currentHash = ComputeSnapshotHash(entries);
if (entries.Count > 0 || fileInfo.Exists && index.Count == 0)
SnapshotRebuilt?.Invoke(this, EventArgs.Empty);
}
// Replace the entire snapshot
ComputeSnapshotHash(entries);
if (snapshotChanged)
{
_logger.LogInformation("Snapshot rebuilt");
SnapshotRebuilt?.Invoke(this, EventArgs.Empty);
}
PersistSnapshotAsync();
return Task.CompletedTask;
}
@@ -318,11 +302,10 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
// Each entry that has not been added to the lookup table is added to the cache
private IEnumerable<FileEntry?> BuildSnapshot(string dir)
{
FileEntry entry;
if (!Directory.Exists(dir)) yield break;
foreach (var file in Directory.EnumerateFiles(dir, "*", SearchOption.AllDirectories))
{
string hash = string.Empty;
var hash = string.Empty;
var ext = Path.GetExtension(file).ToLowerInvariant();
if (!(_options.ArchiveExtensions.Contains(ext) || _options.RomExtensions.Contains(ext)))
@@ -337,6 +320,15 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
var titles = new List<(string, long, NcaMetadataWithHash)>();
if (_options.RomExtensions.Contains(ext))
{
var fileInfo = new FileInfo(file);
var ncaMetadataWithHash = fileInfo.GetNcaMetadataWithHash();
if (ncaMetadataWithHash != null)
{
//var titleInfo = _titleDatabaseService.GetAsync(ncaMetadataWithHash.TitleId).Result;
var fileEntryFromFileName = new FileEntry(file, fileInfo.Length, ncaMetadataWithHash.Hash, [ncaMetadataWithHash]);
AddToSnapshotAsync(fileEntryFromFileName);
yield return fileEntryFromFileName;
}
using var nspStream = File.OpenRead(file);
hash = ComputeFirstStreamHash(nspStream);
@@ -406,72 +398,77 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
}
}
}
private string ComputeFirstStreamHash(Stream nspStream)
private async Task ValidateSnapshotAsync(CancellationToken cancellationToken = default)
{
return _nspExtractor.ExtractHashFromStream(nspStream);
await Task.CompletedTask;
}
private string ComputeFirstStreamHash(Stream nspStream) => _nspExtractor.ExtractHashFromStream(nspStream);
private void UpdateSnapshot() => BuildSnapshotAsync();
IEnumerable<FileEntry> GetEntries()
private IEnumerable<FileEntry> GetEntries()
{
foreach (var snapshotEntry in _cache)
{
_sizeLookup.TryGetValue(snapshotEntry.Value.Hash, out var size);
var fileEntry = new FileEntry(snapshotEntry.Key, snapshotEntry.Value.Size, snapshotEntry.Value.Hash, snapshotEntry.Value.NcaMetadataWithHash);
yield return fileEntry;
}
foreach (var kv in _cache)
yield return new FileEntry(kv.Key, kv.Value.Size, kv.Value.Hash, kv.Value.NcaMetadataWithHash);
}
private Task PersistSnapshotAsync()
{
if (_debouncerCache.TryGetValue(_jsonPath, out var value))
if (_debouncerCache.TryGetValue(_jsonPath, out _))
{
_logger.LogInformation("Sliding debounce in progress, skipping snapshot persistence");
return Task.CompletedTask;
}
}
var entries = GetEntries().ToList();
var newHash = ComputeSnapshotHash(entries);
var snapshot = GetSnapshot();
var entries = GetEntries();
var fileEntries = entries.ToList();
var newHash = ComputeSnapshotHash(fileEntries);
if (snapshot.Hash == newHash) return Task.CompletedTask;
_logger.LogInformation("Snapshot hash changed persisting new snapshot");
using var debouncedPersistence = _debouncerCache.CreateEntry(_jsonPath);
debouncedPersistence.SlidingExpiration = TimeSpan.FromMilliseconds(DebounceMs);
debouncedPersistence.Value = fileEntries;
debouncedPersistence.PostEvictionCallbacks.Add(new PostEvictionCallbackRegistration
{
EvictionCallback = (key, entriesCallback, reason, state) =>
var cancellationTokenSource = new CancellationTokenSource();
using var cacheEntry = _debouncerCache.CreateEntry(_jsonPath)
.AddExpirationToken(new CancellationChangeToken(cancellationTokenSource.Token))
.SetValue(entries)
.SetOptions(new MemoryCacheEntryOptions
{
if (entriesCallback is IEnumerable<FileEntry> entriesToPersist && key is string filePath)
PostEvictionCallbacks =
{
if (_snapshotFileSemaphore.Wait(SnapshotFileLockTimeout))
new PostEvictionCallbackRegistration
{
if (IsFileLocked(filePath))
EvictionCallback = (key, value, reason, state) =>
{
_logger.LogInformation("File {FilePath} is locked, skipping snapshot persistence", filePath);
if (!(reason == EvictionReason.Expired || reason == EvictionReason.TokenExpired))
return;
var filePath = (string)key;
if (_snapshotFileSemaphore.Wait(SnapshotFileLockTimeout))
{
try
{
if (IsFileLocked(filePath))
{
_logger.LogInformation("File {FilePath} is locked, skipping snapshot persistence", filePath);
}
else
{
File.WriteAllText(filePath, JsonSerializer.Serialize(value, _jsonSerializerOptions));
SnapshotRebuilt?.Invoke(this, EventArgs.Empty);
}
}
finally
{
_snapshotFileSemaphore.Release();
}
}
}
else
{
File.WriteAllText(filePath,
JsonSerializer.Serialize(entriesToPersist, _jsonSerializerOptions));
_snapshotFileSemaphore.Release();
_logger.LogInformation("Persisted snapshot");
SnapshotRebuilt?.Invoke(this, EventArgs.Empty);
}
}
else
{
_logger.LogInformation("Failed to persist file {FilePath} due to timeout", filePath);
}
}
}
});
});
cancellationTokenSource.CancelAfter(TimeSpan.FromMilliseconds(DebounceMs));
return Task.CompletedTask;
}
private static string ComputeHash(string filePath)
{
using var sha = SHA256.Create();
@@ -489,6 +486,8 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
}
/// <summary>
/// From filesystem cache, load each entry and build the lookups
/// Check for duplicate hashes
/// Check for nonexistent entries against filesystem
/// </summary>
/// <returns></returns>
private Dictionary<string, FileEntry> LoadSnapshotIndex()
@@ -509,17 +508,40 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
_logger.LogWarning("Duplicate hash found in snapshot: {Hash}, {OldPath}, {newPath}", fileEntry.Hash, value, fileEntry.Path);
}
var nspOrArchivePath = fileEntry.Path.Split(ArchivePathSeparator)[0];
if (!File.Exists(nspOrArchivePath))
{
_logger.LogWarning("Nonexistent entry found: {Path}", fileEntry.Path);
continue;
}
var fileContainedInRootDirectories = false;
foreach (var optionsRootDirectory in _options.RootDirectories)
{
if (fileEntry.Path.StartsWith(optionsRootDirectory))
{
fileContainedInRootDirectories = true;
break;
}
}
if (!fileContainedInRootDirectories)
{
_logger.LogInformation("Entry {Path} is not contained in any root directory", fileEntry.Path);
continue;
};
if (_options.RomExtensions.Contains(Path.GetExtension(fileEntry.Path)))
{
if (fileEntry.Path.Contains(ArchivePathSeparator))
{
var filename = fileEntry.Path.Split(ArchivePathSeparator)[0];
_cache[fileEntry.Path] = new SnapshotEntry(fileEntry.Path, fileEntry.Hash, fileEntry.Size, fileEntry.Titles);
_cache[fileEntry.Path] = new SnapshotEntry(fileEntry.Path, fileEntry.Hash, fileEntry.Size, fileEntry.Titles!);
_archiveLookup[filename] = fileEntry.Path;
}
else
{
_cache[fileEntry.Path] = new SnapshotEntry(fileEntry.Path, fileEntry.Hash, fileEntry.Size, fileEntry.Titles);
_cache[fileEntry.Path] = new SnapshotEntry(fileEntry.Path, fileEntry.Hash, fileEntry.Size, fileEntry.Titles!);
fileEntries.TryAdd(fileEntry.Path, fileEntry);
_hashCache[fileEntry.Hash] = fileEntry.Path;
// ReSharper disable once ConditionIsAlwaysTrueOrFalseAccordingToNullableAPIContract
@@ -531,6 +553,7 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
}
}
}
_logger.LogInformation("Loaded snapshot index {Count} entries", fileEntries.Count);
return fileEntries;
}
catch (ArgumentException e)
@@ -540,7 +563,6 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
}
}
public void RebuildSnapshot()
{
// 1️⃣ Flush the old inmemory snapshot
@@ -658,10 +680,12 @@ public sealed class SnapshotService : IDisposable, ISnapshotService, IHostedServ
_logger.LogInformation("Starting snapshot service");
_ = Task.Run(async () =>
{
await ValidateSnapshotAsync(cancellationToken);
await BuildSnapshotAsync();
await PersistSnapshotAsync();
}, cancellationToken); // initial scan
new Timer(_ => DebounceElapsed(), null, Timeout.Infinite, Timeout.Infinite);
/*var timer = new Timer(_ => DebounceElapsed(), null, Timeout.Infinite, Timeout.Infinite);*/
await Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken)
@@ -24,9 +24,7 @@ public sealed class TitleDatabaseService : IHostedService
private readonly IOptionsMonitor<TitleDbOptions> _options;
private readonly ILogger<TitleDatabaseService> _logger;
private readonly IHttpClientFactory _httpFactory;
private readonly INSPExtractor _nspExtractor;
private readonly string _cacheFolder; // Where the JSON is cached.
private readonly List<string> _rootDirectories; // directories that contain game files
private readonly IMemoryCache _cache;
private readonly ISnapshotService _snapshotService;
@@ -50,7 +48,6 @@ public sealed class TitleDatabaseService : IHostedService
/// directories that contain the NSP files.
/// </summary>
public TitleDatabaseService(
IConfiguration configuration,
IOptionsMonitor<TitleDbOptions> options,
ILogger<TitleDatabaseService> logger,
ISnapshotService snapshotService,
@@ -62,11 +59,10 @@ public sealed class TitleDatabaseService : IHostedService
_logger = logger;
_snapshotService = snapshotService;
_httpFactory = httpFactory;
_nspExtractor = nspExtractor;
_cache = cache;
_cacheFolder = Path.Combine(AppContext.BaseDirectory, "titledb-cache");
_rootDirectories = new List<string>
_cacheFolder = Path.Combine(AppContext.BaseDirectory, "data", "titledb-cache");
new List<string>
{
// You can extend this list it is the set of directories that
// are scanned when the service starts up.
+16 -5
View File
@@ -20,14 +20,23 @@
<Content Include="..\.dockerignore">
<Link>.dockerignore</Link>
</Content>
<Content Update="appsettings.Development.json">
<DependentUpon>appsettings.json</DependentUpon>
</Content>
<Content Remove="obj\**" />
<AdditionalFiles Include="..\libhac\src\LibHac\bin\Release\net8.0\LibHac.dll">
<AdditionalFiles Include="..\Dependencies\LibHac.dll">
<Link>LibHac.dll</Link>
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</AdditionalFiles>
<Content Update="Config\appsettings.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</Content>
<Content Update="Config\appsettings.development.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</Content>
<Content Update="Config\appsettings.local.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</Content>
<Content Update="Config\appsettings.production.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</Content>
</ItemGroup>
<ItemGroup>
@@ -49,6 +58,8 @@
<ItemGroup>
<EmbeddedResource Remove="obj\**" />
<EmbeddedResource Include="appsettings.default.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</EmbeddedResource>
</ItemGroup>
</Project>
+2 -2
View File
@@ -1,6 +1,6 @@
@TinfoilVibeServer_HostAddress = http://localhost:5253
@TinfoilVibeServer_HostAddress = http://tinfoil.localhost/
GET {{TinfoilVibeServer_HostAddress}}/weatherforecast/
GET {{TinfoilVibeServer_HostAddress}}/
Accept: application/json
###
@@ -110,7 +110,7 @@ public static class FileSystemExtensions
/// <exception cref="ArgumentException">Thrown if <paramref name="path"/> is empty or contains only whitespace.</exception>
/// <exception cref="UnauthorizedAccessException">Thrown if the caller does not have permission.</exception>
/// <exception cref="IOException">Thrown if a file exists at the target path or the directory cannot be created.</exception>
public static void EnsureDirectoryExists(string path)
public static void EnsureDirectoryExists(string? path)
{
if (path is null)
throw new ArgumentNullException(nameof(path));
@@ -1,7 +1,7 @@
using System.Globalization;
using System.Text.RegularExpressions;
namespace TinfoilVibeServer.Models;
namespace TinfoilVibeServer.Utilities;
public static class IdHelper
@@ -0,0 +1,30 @@
using System.Security.Cryptography;
using System.Text;
using System.Text.RegularExpressions;
using LibHac.Ncm;
using TinfoilVibeServer.Models;
namespace TinfoilVibeServer.Utilities;
public static class NcaMetadataWithHashHelper
{
public static NcaMetadataWithHash? GetNcaMetadataWithHash(this FileInfo fileInfo)
{
var match = Regex.Match(fileInfo.Name, @"^(.+)\[(\w{16})\]\[v(\d{1,7})\]\[(\w+).*\]\.nsp$");
if (!match.Success) return null;
var titleId = match.Groups[2].Value;
var applicationTitle = match.Groups[1].Value.Trim();
var version = int.Parse(match.Groups[3].Value) / 0x10000;
var nspType = match.Groups[4].Value.ToLowerInvariant() switch
{
"base" => ContentMetaType.Application,
"update" => ContentMetaType.Patch,
"dlc" => ContentMetaType.AddOnContent,
_ => ContentMetaType.Patch
};
var bytes = Encoding.UTF8.GetBytes(fileInfo.FullName);
var hash = SHA256.HashData(bytes);
return new NcaMetadataWithHash(titleId, applicationTitle, version, nspType, Convert.ToBase64String(hash));
}
}
@@ -25,7 +25,7 @@ public sealed class SeekableBufferedStream : Stream
}
private readonly List<BufferBlock> _blocks = new();
private readonly long _specifiedLength = 0;
private readonly long _specifiedLength;
private long _bufferedLength; // total number of bytes buffered so far
private long _position; // current logical position in the stream
private bool _eof; // true when the source stream has been exhausted
@@ -66,7 +66,14 @@ public sealed class SeekableBufferedStream : Stream
}
base.Dispose(disposing);
}
// SeekableBufferedStream.cs Add IAsyncDisposable support
public override async ValueTask DisposeAsync()
{
Dispose(true);
await Task.CompletedTask;
}
#endregion
#region helpers
@@ -244,6 +251,7 @@ public sealed class SeekableBufferedStream : Stream
bytesRead += toCopy;
}
await Task.CompletedTask;
return bytesRead;
}
@@ -1,9 +0,0 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
},
"RootDirectories": [ "D:\\Cloud\\Git\\TinfoilWebServer\\TinfoilWebServer.Test\\data", "Z:\\imgs\\roms\\Switch" ]
}
@@ -0,0 +1,39 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
},
"AllowedHosts": "*",
"CredentialsFile": "/app/data/credentials.json",
"FingerprintsFile": "/app/data/fingerprints.json",
"BlacklistFile": "/app/data/blacklist.json",
"MaxFailedAttempts": 5,
"Snapshot" : {
"RootDirectories": [ ],
"ArchiveExtensions": [ ".zip", ".rar", ".7z" ],
"RomExtensions": [ ".xci", ".nsp", ".xcz" ],
"CacheTtl": 60,
"SnapshotFile": "/app/data/snapshot.json",
"SnapshotBackupFile": "/app/data/snapshot.bak"
},
"NSPExtractor": {
"KeyFile": "/app/config/prod.keys"
},
"IndexBuilder": {
"ApiBaseUrl": "http://tinfoil.localhost:8080",
"IndexDirectories": [
"https://url1",
"sdmc:/url2",
"http://url3"
],
"Success" : "Welcome to Tinfoil Vibe Server!"
},
"TitleDb": {
"CountryCode": "AU",
"Language": "en",
"TtlSeconds" : 90
}
}
-39
View File
@@ -1,39 +0,0 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
},
"AllowedHosts": "*",
"KeySetFile": "prod.keys",
"CredentialsFile": "credentials.json",
"FingerprintsFile": "fingerprints.json",
"BlacklistFile": "blacklist.json",
"MaxFailedAttempts": 5,
"Snapshot" : {
"RootDirectories": [ "Z:\\downloads\\roms\\switch", "Z:\\imgs\\roms\\Switch" ],
"ArchiveExtensions": [ ".zip", ".rar", ".7z" ],
"RomExtensions": [ ".xci", ".nsp", ".xcz" ],
"CacheTtl": 60,
"SnapshotFile": "index.tfl",
"SnapshotBackupFile": "snapshot.bin"
},
"IndexBuilder": {
"ApiBaseUrl": "http://tinfoil.localhost:80"
},
"TitleDb": {
"CountryCode": "AU",
"Language": "en",
"TtlSeconds" : 90,
"SnapshotFile" : "snapshot.json"
},
"IndexDirectories": [
"https://url1",
"sdmc:/url2",
"http://url3"
],
"Success" : "Welcome to Tinfoil Vibe Server!"
}
@@ -0,0 +1,8 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
}
}
+39
View File
@@ -0,0 +1,39 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
},
"AllowedHosts": "*",
"CredentialsFile": "/app/data/credentials.json",
"FingerprintsFile": "/app/data/fingerprints.json",
"BlacklistFile": "/app/data/blacklist.json",
"MaxFailedAttempts": 5,
"Snapshot" : {
"RootDirectories": [ ],
"ArchiveExtensions": [ ".zip", ".rar", ".7z" ],
"RomExtensions": [ ".xci", ".nsp", ".xcz" ],
"CacheTtl": 60,
"SnapshotFile": "/app/data/snapshot.json",
"SnapshotBackupFile": "/app/data/snapshot.bak"
},
"NSPExtractor": {
"KeyFile": "/app/config/prod.keys"
},
"IndexBuilder": {
"ApiBaseUrl": "http://tinfoil.localhost:8080",
"IndexDirectories": [
"https://url1",
"sdmc:/url2",
"http://url3"
],
"Success" : "Welcome to Tinfoil Vibe Server!"
},
"TitleDb": {
"CountryCode": "AU",
"Language": "en",
"TtlSeconds" : 90
}
}
@@ -0,0 +1,9 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"System": "Information",
"Microsoft": "Information"
}
}
}
View File
View File
@@ -1,4 +1,5 @@
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Hosting.Internal;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Moq;
using TinfoilVibeServer.Authentication;
@@ -20,7 +21,8 @@ namespace TinfoilVibeServerTest.Tests
_loggerMock = new Mock<ILogger<AuthStore>>();
// Assume Settings is static and can be patched for tests
MockConfigManager = new Mock<ConfigManager>();
_authStore = new AuthStore(_loggerMock.Object, MockConfigManager.Object);
var env = new Mock<HostingEnvironment>();
_authStore = new AuthStore(_loggerMock.Object, MockConfigManager.Object, env.Object);
}
public Mock<ConfigManager> MockConfigManager { get; set; }
@@ -39,7 +41,7 @@ namespace TinfoilVibeServerTest.Tests
var fprs = _authStore.Fingerprints.Count;
// Assert
Assert.That(users, Is.GreaterThan(0), "At least one user must be loaded");
//Assert.That(users, Is.GreaterThan(0), "At least one user must be loaded");
Assert.That(fprs, Is.GreaterThanOrEqualTo(0));
_loggerMock.Verify(
@@ -59,7 +61,7 @@ namespace TinfoilVibeServerTest.Tests
var newUser = "newuser";
var ip = "127.0.0.1";
var password = "";
var uid = null as int?;
var uid = "";
// Act
var result = _authStore.TryValidate(newUser, password, uid, ip, out var cred);
@@ -31,7 +31,7 @@ namespace TinfoilVibeServerTest.Tests
_middleware = new BasicAuthMiddleware(_next);
}
private HttpContext CreateContext(string authHeader = "", string ip = "127.0.0.1", int? uid = null)
private HttpContext CreateContext(string authHeader = "", string ip = "127.0.0.1", string uid = "")
{
var ctx = new DefaultHttpContext();
ctx.Connection.RemoteIpAddress = IPAddress.Parse(ip);
@@ -54,6 +54,7 @@ namespace TinfoilVibeServerTest.Tests
{
// Arrange
var ctx = CreateContext();
ctx.Request.Path = new PathString("/");
// Act
await _middleware.InvokeAsync(ctx, _authMock.Object, _loggerMock.Object);
@@ -73,7 +74,7 @@ namespace TinfoilVibeServerTest.Tests
{
// Arrange
var ctx = CreateContext("Basic dXNlcjpwYXNz");
ctx.Request.Path = new PathString("/");
_authMock.Setup(a => a.IsIPBlacklisted("127.0.0.1")).Returns(true);
// Act
@@ -90,7 +91,7 @@ namespace TinfoilVibeServerTest.Tests
// Arrange
var user = "alice";
var pw = "secret";
var uid = 1234;
var uid = "1234";
var header = $"Basic {Convert.ToBase64String(Encoding.ASCII.GetBytes($"{user}:{pw}"))}";
var ip = "127.0.0.1";
@@ -3,6 +3,7 @@ using System.IO;
using System.Threading.Tasks;
using LibHac.Ncm;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Hosting.Internal;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Moq;
@@ -44,6 +45,7 @@ namespace TinfoilVibeServerTest.Tests
_loggerMock = new Mock<ILogger<SnapshotService>>();
_archiveHander = new Mock<IArchiveHandler>();
_nspExtractorMock = new Mock<INSPExtractor>();
var hostEnv = new Mock<HostingEnvironment>();
var memoryCacheOptions = Options.Create(new MemoryCacheOptions());
_memoryCache = new MemoryCache(memoryCacheOptions);
@@ -52,7 +54,8 @@ namespace TinfoilVibeServerTest.Tests
_nspExtractorMock.Setup(extractor => extractor.ExtractFromStream(It.IsAny<Stream>())).Returns(
new NcaMetadataWithHash(titleId: "0000000000000000","0000000000000000", version: 1, ContentMetaType.Application, "HASH"));
//Settings.RootDirs = new List<string> { "TestData/Root1", "TestData/Root2" };
_service = new SnapshotService(_memoryCache, _mockOptions.Object, _nspExtractorMock.Object, _archiveHander.Object, _loggerMock.Object);
_service = new SnapshotService(_memoryCache, _mockOptions.Object, _nspExtractorMock.Object, _archiveHander.Object, _loggerMock.Object, hostEnv.Object);
}
[TearDown]
+8
View File
@@ -0,0 +1,8 @@
# docker-compose.override.yml
services:
tinfoilvibeserver:
environment:
- LOG_LEVEL=error
- APP_MODE=production
ports:
- "8080:8080" # expose on host port 80
+19 -10
View File
@@ -1,13 +1,22 @@
services:
consoleapp1:
image: consoleapp1
build:
context: .
dockerfile: ConsoleApp1/Dockerfile
version: "3.9"
services:
tinfoilvibeserver:
image: tinfoilvibeserver
build:
context: .
dockerfile: TinfoilVibeServer/Dockerfile
context: TinfoilVibeServer
dockerfile: Dockerfile
image: gitea.ecenshu.net/ecenshu/tinfoilvibeserver:latest
container_name: tinfoilvibeserver
restart: unless-stopped
user: "1000:1000"
env_file:
- .env
environment:
- ASPNETCORE_ENVIRONMENT=Production # .NETspecific
- LOG_LEVEL=${LOG_LEVEL} # just a doublecheck, uses .env value
volumes:
- ./data:/app/data
- ./config:/app/config
ports:
- ":80"