inference_layer
github.com/amithathreya/inference_layer
8c88c66mastersynced 2w ago
Code HealthView report
Average health
8.9/10
Excellent
Hotspot health
8.2/10
Excellent
CostsView costs →
What this repo's indexing and doc generation cost, per sync.
Open the Costs page →Attention Needed11
KVCacheManager/kvcache.pyUngoverned Hotspot
High-churn file with no governing decision
KVCacheManager/RequestStruct.pyKnowledge Silo
100% single-owner concentration
KVCacheManager/kvcache.pyKnowledge Silo
100% single-owner concentration
KVCacheManager/sequence.pyKnowledge Silo
100% single-owner concentration
inference_scheduler/init.pyKnowledge Silo
100% single-owner concentration
No commit history
Commit activity appears once git history is indexed.
Recent DecisionsView all
1. Initialize the LLaMA model Note: llama-cpp-python requires GGUF formatted model files, not raw HFproposed
· code commentUse smaller quantization if llama-server OOMsproposed
· readme miningRecommend Q4_K_M quantization for CPU-only setupsproposed
· readme miningAdd authentication and rate limitingproposed
· readme miningAdd SSE streaming for UIproposed
· readme miningImplement KV cache management via PagedAttentionproposed
· readme miningTop Hotspots1 of 1View all
Commit Activity
Feature Fix Refactor Dependency