NVIDIA Researchers Improve Transformer Variants with New Projection | Akshay Pachaar님이 토픽에 대해 올림

NVIDIA researchers built a new transformer variant. One small change to the layers made: - decoding 1.7x faster - long-reasoning accuracy up 6.5 points Every attention layer computes Q, K, and V. NVIDIA’s tweak adds a fourth projection that predicts what the next layer will need. To see why, look at long-context inference today. Sparse attention scores the KV cache in blocks, keeps the top-k, and attends only to those. That cuts compute and bandwidth, but two problems remain. > First, the KV cache still grows with every generated token. At 100K+ context it no longer fits in GPU memory and gets offloaded to CPU RAM. Every layer must then copy its selected blocks back, and the GPU sits idle during that copy, at every layer of every decode step. > Second, the selection step itself is not free. Standard selectors score every candidate block with every query head in a GQA group (grouped-query attention, where several query heads share one KV head), then softmax and sum across the group. Decode attention is cheap with one query token, but this selection cost grows with context length. Both problems trace back to the same design, i.e., the attention query drives the block selection. Q only exists once its layer is already running, so it is too late to fetch anything early, and the query drags its multi-head layout into selection. A recent paper from NVIDIA and MIT called SparDA breaks this coupling. Each layer now emits four projections instead of three: ↳ Q, K, V, and a Forecast. The Forecast from layer L predicts which KV blocks layer L+1 will need, and layer L+1’s own query then attends over those blocks. Since the next layer’s block set is known while the current layer is still computing, the runtime fetches those blocks on a separate CUDA stream, overlapping the copy with compute. And since the Forecast is separate from the query, it needs one head per GQA group, which removes the per-query-head scoring loop and skips the softmax entirely. DeepSeek did something similar in DSA, where a small indexer picks important tokens. SparDA applies it to blocks and adds the prefetch angle. The cost is small. The Forecast adds 33.5M parameters on an 8B model (0.41%), and only those projections are trained, using a KL loss that matches the original selector’s block distribution. On MiniCPM4.1-8B and NOSA-8B, accuracy matches or beats the sparse baseline, with NOSA-8B gaining +6.5 on long reasoning. Prefill runs up to 1.25x faster and decode up to 1.7x faster than the sparse offload baseline. There’s one more benefit. Because prefetch hides the offload cost, most of the KV cache can live in CPU RAM, and the freed GPU memory fits much bigger batches, pushing decode throughput up to 5.3x over the non-offload sparse baseline. That said, the lookahead only pays off during decode with CPU offload. During prefill all keys already live on the GPU, so the gain comes purely from cheaper selection. Link to the paper in the first comment.

댓글 4

더 보기

댓글을 보거나 남기려면 로그인

관련성이 더 높은 업데이트

  • 106% of cuBLAS. At first glance, that sounds impossible. How can a custom GEMM kernel outperform NVIDIA’s highly optimized cuBLAS library? The answer isn’t a magical optimization. It’s the cumulative effect of exploiting every major architectural feature introduced in NVIDIA Blackwell, while redesigning the kernel around the hardware instead of treating the GPU like a faster CPU. The optimization journey starts with a conventional BF16 Tensor Core GEMM using Tensor Memory Accelerator (TMA) to move tiles from HBM into shared memory before Tensor Cores perform matrix multiply-accumulate operations. This baseline reaches only about 40% of cuBLAS, not because Tensor Cores are slow, but because they spend much of their time waiting for memory and synchronization. The first breakthrough comes from 2-CTA MMA. Previous architectures restricted matrix multiply operations to a single Cooperative Thread Array (CTA). Blackwell’s 5th-generation Tensor Cores allow two CTAs to cooperate on one larger MMA instruction. Instead of performing two independent 128×256 operations, two neighboring CTAs execute one 256×256 operation. This removes redundant memory loads, increases arithmetic intensity from roughly 85 FLOPs/byte to 128 FLOPs/byte, and dramatically improves data reuse. The GPU performs more computation for every byte fetched from memory. The kernel then adopts warp specialization. Rather than assigning identical responsibilities to every warp, different warps become specialists. Producer warps exclusively issue TMA memory transfers, consumer warps issue MMA instructions, and separate epilogue warps handle output writes. This separation allows memory movement, Tensor Core execution, and result storage to overlap continuously, increasing overall streaming multiprocessor utilization. Another major improvement is the transition to persistent kernels. Traditional GEMM launches a new CUDA kernel for every output tile, repeatedly paying launch overhead and initialization costs. Persistent kernels launch only enough thread blocks to occupy the GPU’s Streaming Multiprocessors, then repeatedly fetch new work internally. Prologue costs are paid once, while Tensor Memory allocation and synchronization structures remain active throughout execution, keeping both Tensor Cores and TMA engines busy almost continuously. The final result is remarkable: a custom Blackwell GEMM kernel that reaches 106% of cuBLAS for selected workloads. More importantly, it demonstrates that future AI performance is no longer driven solely by faster Tensor Cores. The largest gains now come from understanding memory hierarchy, scheduling, synchronization, asynchronous execution, cache locality, persistent execution models, and hardware-aware software design. In the Blackwell era, optimizing AI workloads is no longer just CUDA programming. It is computer architecture engineering in software. 댓글을 보거나 남기려면 로그인
  • If a computer with large 𝘂𝗻𝗶𝗳𝗶𝗲𝗱 𝗺𝗲𝗺𝗼𝗿𝘆 (Mac Mini) is so much cheaper than Nvidia’s 𝗱𝗮𝘁𝗮 𝗰𝗲𝗻𝘁𝗲𝗿 𝗚𝗣𝗨𝘀 (H100, H200, B200), why can’t we just build a cluster of Mac Mini to serve AI workloads? One Mac Mini is much less powerful than one H100, but it’s also way cheaper. Can’t we just buy more of them and achieve better performance per unit of price? Unfortunately, it’s not that simple. 📉 𝗠𝗲𝗺𝗼𝗿𝘆 𝗯𝗮𝗻𝗱𝘄𝗶𝗱𝘁𝗵 LLM inference is bandwidth-bound, not compute-bound. For every token generated, the GPU has to stream the entire model’s weights out of memory. An M4 Pro Mac Mini has ~273 GB/s of memory bandwidth. An H100 has ~3.35 TB/s of HBM3, and a B200 pushes ~8 TB/s. More Mac Mini doesn’t fix this per-unit ceiling. 🔗 𝗜𝗻𝘁𝗲𝗿𝗰𝗼𝗻𝗻𝗲𝗰𝘁 𝗯𝗼𝘁𝘁𝗹𝗲𝗻𝗲𝗰𝗸 Inference on large models increasingly means splitting a single model across multiple chips (tensor/pipeline parallelism), which requires constant, fast communication between them. Nvidia GPUs talk to each other over NVLink at ~900 GB/s per GPU. Mac Mini has to talk over Thunderbolt 5 at ~15 GB/s or Ethernet (even slower). More Mac Mini compounds the problem. ⚡ 𝗧𝗵𝗿𝗼𝘂𝗴𝗵𝗽𝘂𝘁 𝗮𝘁 𝘀𝗰𝗮𝗹𝗲 (𝗯𝗮𝘁𝗰𝗵𝗶𝗻𝗴) This is where the FLOPS matters. Serving many users at once means batching requests, and batched throughput depends on raw compute. An H100 delivers ~1,000 TFLOPS of dense FP16/BF16 compute. A Mac Mini’s GPU cores are nowhere close. They’re built for single-user, on-device workloads. Again, more Mac Mini doesn’t solve this problem. 🧰 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗲𝗰𝗼𝘀𝘆𝘀𝘁𝗲𝗺 Nvidia’s stack (CUDA, cuDNN, TensorRT, NCCL) is what most production inference/training frameworks (vLLM, TensorRT-LLM, DeepSpeed, Megatron) are optimized around. Multi-node GPU clusters have years of tooling for scheduling, sharding, and fault tolerance. Apple Silicon inference relies on MLX or llama.cpp, which are excellent for local, single-machine use but aren’t as mature. · · · 𝗪𝗵𝗲𝗿𝗲 𝗹𝗮𝗿𝗴𝗲 𝘂𝗻𝗶𝗳𝗶𝗲𝗱 𝗺𝗲𝗺𝗼𝗿𝘆 𝘄𝗼𝗿𝗸𝘀 It’s just a different use case. A 64GB+ unified memory pool lets you fit a large model on a single box for local/offline inference, prototyping, or single-user workloads, without worrying about the CPU-GPU memory split. It’s a great value for “I want to run a big model on my desk.” 댓글 27 댓글을 보거나 남기려면 로그인
  • I reproduced an NVIDIA paper “Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization” (Ghita et al., arXiv:2607.24518). The idea: take an n×n dependence matrix between financial instruments and factorize it as S ≈ H·Hᵀ with H non-negative. The columns of H become interpretable risk factors; each row is a soft cluster assignment of one instrument. They study two inputs - the familiar correlation matrix, and the tail pairwise dependence matrix (TPDM) from Extreme Value Theory, which captures co-crash structure that correlation misses entirely. The trick that makes it a GPU problem: a naive implementation of ‖S − HHᵀ‖² materializes n×n intermediates (~20n² bytes). The paper’s trace-identity reformulation eliminates all of them, cutting memory to ~4n² bytes and reducing every solver iteration to two dense cuBLAS matrix products. That one algebraic identity is why they reach n ≈ 100,000 on a single GPU. I reimplemented it in CuPy on an RTX PRO 5000 Blackwell laptop GPU and ran their benchmark: a planted block factor model, both matrix types, their AdaGrad solver against classical multiplicative updates, and their strict three-part convergence gate. Results from my machine: → A converged factorization of a 4,000-instrument universe: 1–3 seconds. → The planted factor structure recovered exactly from the permuted matrix — purity 1.000 on correlation inputs. → Throughput sweep, 200 iterations: GPU 1.2× faster than CPU at n=1,000, 4.8× at n=2,000, 11.5× at n=4,000, 29.5× at n=8,000. The CPU curve grows as O(n²k); the GPU curve barely moves. → At ~4n² bytes, a 24 GB card clears roughly 70,000 instruments, full-universe scale for an equity risk model. Why it matters for quant work: factor extraction over an entire book stops being an overnight batch job. At seconds per solve, re-estimating your factor structure becomes something you can do intraday, on hardware that fits in a backpack. Honest caveats: benchmark matrices are synthetic (the paper’s own construction), the AdaGrad step size needed re-tuning on my benchmark (η=0.05 vs their 2.0 - step size is spectrum-dependent), and my demo sizes sit below the scale where the paper shows multiplicative updates start failing. Educational reproduction, not investment advice. Full experiment report with methodology, tables, and figures in the comments/attached. What’s the largest dependence matrix you’ve had to factorize and how long did it take? #DellTech #NVIDIA #DellProPrecision Logan Lawler Cindy Olivo 댓글 1 댓글을 보거나 남기려면 로그인
  • NVIDIA merged our change into TensorRT-LLM this morning. Same GPU, same model, same answers — 40% more throughput and nearly half the memory. Nemotron, NVIDIA’s flagship open model family, on NVIDIA’s own inference server, on an NVIDIA GPU — their stack end to end, already fast, reviewed and merged by their engineers. On an A100: → Throughput: 1,203 → 1,690 tokens/sec (+40%) → Time per token: 23.6 → 16.5 ms (−30%) → Weight memory: 58.9 → 31.5 GB (−27 GB) Lossless on gsm8k. As a control, Qwen3.6 — a gated MoE where it should change nothing — measured exactly that. Unusually, all three moved together — normally you trade one for another. And 27 GB is a third of the card, handed back to serving. The kernels were ahead of the plumbing. They had handled this model’s layout the whole time, and the newer compression formats already used it. It was the older INT8 path that still declined the model on shape grounds rather than capability grounds. We brought it to parity: two files, no new kernels, nothing behind a flag, covering the whole non-gated family. The wider point matters more than the PR. An A100 has no FP8, so INT8 is how you compress a model on one — which means this lands on cards that are already installed, not on the ones on order. When throughput runs short, the instinct is to buy more GPUs. But software is rarely finished. The hardware you already own usually has room left in it, and unlocking it costs no capital. That’s where we put our effort, and the headroom keeps being larger than teams expect. That’s what we automate. Last week Jeff Dean and his co-founders launched Discovery Loop, on the thesis that automating the experimental loop — propose, run, measure, iterate — unlocks progress in engineering. I think that’s right, and we run that loop against production systems rather than research: inference stacks, speech models, quant risk libraries and Monte Carlo simulations, trading hot paths, the SQL underneath everything — the models themselves. Anything with a number someone already cares about. They also intend to be their own first customers. So do we: we serve LLMs to run Artemis, on hardware we pay for, so its inference bill is our inference bill. That’s why we were in this code. In Artemis we call it Discovery. An evolutionary search — work we published before LLMs existed — with models now doing the proposing. What a model can’t tell you is whether its change really worked on your hardware, under your load, without breaking anything. So every candidate is built, run on the real target and measured: here, 21 candidate changes across 15 experiments, each scored three times. Most fail. You only ever see the one that survived — and the last mile was real engineering: Dorian Donaj Magasic drove the run and carried the change upstream, on the Discovery engine Paul Brookes and the team have been building. Our second merged PR in a frontier lab’s repo this summer. Next: the 4.9× kernel we found inside OpenAI’s Whisper. 댓글 3 댓글을 보거나 남기려면 로그인
  • AI’s next frontier isn’t simply about building bigger models or deploying more GPUs. It’s about optimization at scale. As AI adoption accelerates, the industry is confronting a fundamental reality: compute, power, infrastructure and capital are not unlimited. If we want AI to reach its full potential, we have to get significantly more performance from the infrastructure we already have. That is why I believe TurinTech AI represents an important part of the future of AI infrastructure. The opportunity is to optimize software and AI workloads so they execute more efficiently — improving performance, reducing unnecessary compute requirements, and ultimately allowing organizations to do more with the same infrastructure. Think about what that means at scale. Across thousands of GPUs, hyperscale data centers and increasingly complex AI workloads, even incremental improvements in efficiency can translate into enormous gains in capacity, economics and energy utilization. We cannot solve the world’s demand for AI simply by continuing to add more GPUs, more data centers and more gigawatts. We also have to make the technology running on that infrastructure dramatically more efficient. The future of AI will be built on two things: more compute and smarter compute. TurinTech is helping build the smarter side of that equation. The AI infrastructure race is just beginning — and optimization will be one of the technologies that determines how far and how fast we can scale. #AI #ArtificialIntelligence #AIInfrastructure #DataCenters #GPUs #Optimization #TurinTech #Compute #EnergyEfficiency NVIDIA merged our change into TensorRT-LLM this morning. Same GPU, same model, same answers — 40% more throughput and nearly half the memory. Nemotron, NVIDIA’s flagship open model family, on NVIDIA’s own inference server, on an NVIDIA GPU — their stack end to end, already fast, reviewed and merged by their engineers. On an A100: → Throughput: 1,203 → 1,690 tokens/sec (+40%) → Time per token: 23.6 → 16.5 ms (−30%) → Weight memory: 58.9 → 31.5 GB (−27 GB) Lossless on gsm8k. As a control, Qwen3.6 — a gated MoE where it should change nothing — measured exactly that. Unusually, all three moved together — normally you trade one for another. And 27 GB is a third of the card, handed back to serving. The kernels were ahead of the plumbing. They had handled this model’s layout the whole time, and the newer compression formats already used it. It was the older INT8 path that still declined the model on shape grounds rather than capability grounds. We brought it to parity: two files, no new kernels, nothing behind a flag, covering the whole non-gated family. The wider point matters more than the PR. An A100 has no FP8, so INT8 is how you compress a model on one — which means this lands on cards that are already installed, not on the ones on order. When throughput runs short, the instinct is to buy more GPUs. But software is rarely finished. The hardware you already own usually has room left in it, and unlocking it costs no capital. That’s where we put our effort, and the headroom keeps being larger than teams expect. That’s what we automate. Last week Jeff Dean and his co-founders launched Discovery Loop, on the thesis that automating the experimental loop — propose, run, measure, iterate — unlocks progress in engineering. I think that’s right, and we run that loop against production systems rather than research: inference stacks, speech models, quant risk libraries and Monte Carlo simulations, trading hot paths, the SQL underneath everything — the models themselves. Anything with a number someone already cares about. They also intend to be their own first customers. So do we: we serve LLMs to run Artemis, on hardware we pay for, so its inference bill is our inference bill. That’s why we were in this code. In Artemis we call it Discovery. An evolutionary search — work we published before LLMs existed — with models now doing the proposing. What a model can’t tell you is whether its change really worked on your hardware, under your load, without breaking anything. So every candidate is built, run on the real target and measured: here, 21 candidate changes across 15 experiments, each scored three times. Most fail. You only ever see the one that survived — and the last mile was real engineering: Dorian Donaj Magasic drove the run and carried the change upstream, on the Discovery engine Paul Brookes and the team have been building. Our second merged PR in a frontier lab’s repo this summer. Next: the 4.9× kernel we found inside OpenAI’s Whisper. 댓글을 보거나 남기려면 로그인
  • Really proud of the recent work we have been carrying out at TurinTech AI! One of the defining challenges in the global AI race is no longer just building more capable frontier models, but serving the latest generation of models efficiently on local hardware, while minimising memory footprint, preserving model quality and maximising inference speed. That is exactly the problem we tackled with NVIDIA’s Nemotron. Optimising NVIDIA’s flagship model, through its TensorRT inference stack, specifically on NVIDIA’s GPU. Model. Runtime. Hardware. Optimised as one system. Take a look below: NVIDIA merged our change into TensorRT-LLM this morning. Same GPU, same model, same answers — 40% more throughput and nearly half the memory. Nemotron, NVIDIA’s flagship open model family, on NVIDIA’s own inference server, on an NVIDIA GPU — their stack end to end, already fast, reviewed and merged by their engineers. On an A100: → Throughput: 1,203 → 1,690 tokens/sec (+40%) → Time per token: 23.6 → 16.5 ms (−30%) → Weight memory: 58.9 → 31.5 GB (−27 GB) Lossless on gsm8k. As a control, Qwen3.6 — a gated MoE where it should change nothing — measured exactly that. Unusually, all three moved together — normally you trade one for another. And 27 GB is a third of the card, handed back to serving. The kernels were ahead of the plumbing. They had handled this model’s layout the whole time, and the newer compression formats already used it. It was the older INT8 path that still declined the model on shape grounds rather than capability grounds. We brought it to parity: two files, no new kernels, nothing behind a flag, covering the whole non-gated family. The wider point matters more than the PR. An A100 has no FP8, so INT8 is how you compress a model on one — which means this lands on cards that are already installed, not on the ones on order. When throughput runs short, the instinct is to buy more GPUs. But software is rarely finished. The hardware you already own usually has room left in it, and unlocking it costs no capital. That’s where we put our effort, and the headroom keeps being larger than teams expect. That’s what we automate. Last week Jeff Dean and his co-founders launched Discovery Loop, on the thesis that automating the experimental loop — propose, run, measure, iterate — unlocks progress in engineering. I think that’s right, and we run that loop against production systems rather than research: inference stacks, speech models, quant risk libraries and Monte Carlo simulations, trading hot paths, the SQL underneath everything — the models themselves. Anything with a number someone already cares about. They also intend to be their own first customers. So do we: we serve LLMs to run Artemis, on hardware we pay for, so its inference bill is our inference bill. That’s why we were in this code. In Artemis we call it Discovery. An evolutionary search — work we published before LLMs existed — with models now doing the proposing. What a model can’t tell you is whether its change really worked on your hardware, under your load, without breaking anything. So every candidate is built, run on the real target and measured: here, 21 candidate changes across 15 experiments, each scored three times. Most fail. You only ever see the one that survived — and the last mile was real engineering: Dorian Donaj Magasic drove the run and carried the change upstream, on the Discovery engine Paul Brookes and the team have been building. Our second merged PR in a frontier lab’s repo this summer. Next: the 4.9× kernel we found inside OpenAI’s Whisper. 댓글을 보거나 남기려면 로그인
  • NVIDIA merged our change into TensorRT-LLM this morning. Same GPU, same model, same answers — 40% more throughput and nearly half the memory. Nemotron, NVIDIA’s flagship open model family, on NVIDIA’s own inference server, on an NVIDIA GPU — their stack end to end, already fast, reviewed and merged by their engineers. On an A100: → Throughput: 1,203 → 1,690 tokens/sec (+40%) → Time per token: 23.6 → 16.5 ms (−30%) → Weight memory: 58.9 → 31.5 GB (−27 GB) Lossless on gsm8k. As a control, Qwen3.6 — a gated MoE where it should change nothing — measured exactly that. Unusually, all three moved together — normally you trade one for another. And 27 GB is a third of the card, handed back to serving. The kernels were ahead of the plumbing. They had handled this model’s layout the whole time, and the newer compression formats already used it. It was the older INT8 path that still declined the model on shape grounds rather than capability grounds. We brought it to parity: two files, no new kernels, nothing behind a flag, covering the whole non-gated family. The wider point matters more than the PR. An A100 has no FP8, so INT8 is how you compress a model on one — which means this lands on cards that are already installed, not on the ones on order. When throughput runs short, the instinct is to buy more GPUs. But software is rarely finished. The hardware you already own usually has room left in it, and unlocking it costs no capital. That’s where we put our effort, and the headroom keeps being larger than teams expect. That’s what we automate. Last week Jeff Dean and his co-founders launched Discovery Loop, on the thesis that automating the experimental loop — propose, run, measure, iterate — unlocks progress in engineering. I think that’s right, and we run that loop against production systems rather than research: inference stacks, speech models, quant risk libraries and Monte Carlo simulations, trading hot paths, the SQL underneath everything — the models themselves. Anything with a number someone already cares about. They also intend to be their own first customers. So do we: we serve LLMs to run Artemis, on hardware we pay for, so its inference bill is our inference bill. That’s why we were in this code. In Artemis we call it Discovery. An evolutionary search — work we published before LLMs existed — with models now doing the proposing. What a model can’t tell you is whether its change really worked on your hardware, under your load, without breaking anything. So every candidate is built, run on the real target and measured: here, 21 candidate changes across 15 experiments, each scored three times. Most fail. You only ever see the one that survived — and the last mile was real engineering: Dorian Donaj Magasic drove the run and carried the change upstream, on the Discovery engine Paul Brookes and the team have been building. Our second merged PR in a frontier lab’s repo this summer. Next: the 4.9× kernel we found inside OpenAI’s Whisper. 댓글 1 댓글을 보거나 남기려면 로그인
  • Teams open nvidia-smi, see 95% GPU utilization, and assume the GPU is working hard for them. It isn’t. Not always. GPU Utilization %: tells you a kernel is running. Says nothing about whether that kernel is doing useful work. Tensor Core Utilization: tells you if you’re actually using the matrix units the GPU was built for. Memory Bandwidth Utilization: tells you if you’re compute-bound or memory-bound. The issue: You can sit at 95% GPU utilization while your tensor cores are nearly idle and the real bottleneck is memory movement. Why: Small batch sizes underuse tensor cores. The GPU looks “busy” but is doing near-scalar work. Attention layers are frequently memory-bandwidth bound, not compute bound. Kernel launch overhead between small ops keeps the GPU “active” without adding FLOPs. Real scenario: Dashboard shows 92% GPU utilization. Inference is still slow. You profile with Nsight and find 60% of time is spent moving KV cache in and out of memory, not computing attention. The GPU is busy waiting, not busy computing. Solution: Don’t trust GPU utilization % alone. Profile with Nsight Systems / Compute. Check tensor core utilization specifically. Fix batching. Too small wastes tensor cores, too large blows out the KV cache. Fuse kernels where you can to cut launch overhead. Move memory-bound ops (long-context attention) to paged KV cache / FlashAttention-style kernels. Most teams optimize the number on the dashboard. Not the number that’s actually costing them. This is the exact loop we run before touching a single line of model code: profile first, guess never. #GPUOptimization #MLOps #MachineLearning #AIInfrastructure #DeepLearning 댓글을 보거나 남기려면 로그인
  • Nvidia lost 589 billion of market value, the largest one-day loss by any company in US market history. The trigger was a claim that a Chinese lab had trained a frontier-competitive model for about 5.6 million. Here's what almost nobody said out loud that week: the number wasn't new. It was published on 26-27 December 2024, in the DeepSeek-V3 technical report -- not R1, the reasoning model that actually shipped a chat app and hit #1 on the US App Store on the 27th. The market didn't learn anything that Monday. It noticed something it had ignored for 32 days. Read the receipt and the headline stops being mysterious: 2.664M GPU-hours pre-training, 119K for context-length extension, 5K post-training -- 2.788M H800 GPU-hours total, at DeepSeek's own assumed rental rate of 2 per hour. 5.6M covers only the official V3 training run, excluding prior research, ablations and failed architectures. SemiAnalysis put the lab’s total server capex near $1.6 billion, across roughly 50,000 Hopper-class GPUs. Both numbers are true. They measure different things — and DeepSeek said so before anyone else did. What actually mattered: weights under an MIT license, the full report, and the DualPipe and DeepEP kernels all published as open repositories. There was no moat to breach because there was no secret to keep. The real change isn’t “AI got cheap.” It’s that the efficiency frontier is now public, itemized and reproducible — and a hardware restriction functioned as an architecture brief. Next time you see a training-cost number, read it the way an engineer reads an invoice: which line does it cover, and which lines does it leave off? #DeepSeek #AIInfrastructure #MachineLearning #LLMTraining #AIEngineering 댓글을 보거나 남기려면 로그인
  • in honor of the Rubin spec sheet coming out, i’ll be doing a series on Rubin’s low level software. starting with a GUIDE on HOW TO ACTUALLY READ AN NVIDIA SPEC SHEET: nvidia’s headline for rubin is 50 petaflops of nvfp4, 5x blackwell. the dense number is 35. the word “sparse” appears nowhere on the spec page. you have to open the pdf datasheet, where footnote 1 ends with five words: “NVFP4 Inference specification is sparse.” FIVE CORE RULES to decode any nvidia spec sheet. rule 1: find the density basis. every headline AI number since a100 assumes 2:4 structured sparsity. in 2020 the dense number came first: “156 TF | 312 TF*”. h100 flipped the order in 2022, then quietly deleted the dense column: “Shown with sparsity. Specifications are one-half lower without sparsity.” blackwell launched at 20 petaflops of fp4 with no dense number on the page, just a two word footnote: “With sparsity.” and the sparse premium isn’t 2x anymore. gb300 is 20 sparse vs 15 dense. rubin is 50 vs 35, and the 35 is labeled “training”. a dense inference number for rubin does not exist anywhere. rule 2: match the precision on both sides of any Nx. a100’s “20x volta” was sparse tf32 tensor ops against v100’s fp32 cuda cores. same unit and same precision, the gain was 2.5x. h100’s “30x” ran fp8 against fp16, on 32 a100s vs 16 h100s, at batch 4 vs 60. blackwell’s “30x h100” was fp4 vs fp8. rubin’s “5x blackwell” compares compressed 50 against dense 10. dense to dense it’s 3.5x, the biggest like-for-like gain in years. but it’s 3.5, not 5. rule 3: ask what “one GPU” means. blackwell is two dies sold as one gpu. at gtc 2025 jensen said that naming “was wrong” and the rubin rack debuted as NVL144, counting dies. at ces 2026 it became NVL72 again, counting packages. same rack, same silicon, both names. the only acknowledgment is an editor’s note retrofitted onto an old blog post. even one b200 has three official fp4 ratings: 20 petaflops grace-paired, 18 on hgx boards, 14 in b100. all sparse, obviously. rule 4: “up to” means one point on a curve. rubin’s “10x tokens per megawatt” chart is real. the 10x arrow is drawn above 200 tokens/sec/user, exactly where blackwell’s curve collapses. the left edge of the same chart reads 1.7x. every generation runs this play: gb300’s “50x vs hopper” is 10x responsiveness times 5x throughput, multiplied together right on the product page. rule 5: the footnotes never lie. rubin’s 200 teraflops of fp64? footnote 3: “Peak performance using Tensor Core-based emulation algorithms.” native fp64 is 33 teraflops, down from blackwell’s 40. and the spec sheet has no power row at all, not for the gpu, not for the rack, so every tokens-per-megawatt denominator is unauditable. none of this is hidden btw! the real numbers are all there, in the fine print. nvidia is rigorous about footnotes precisely because the headlines are doing so much work. 댓글 5 댓글을 보거나 남기려면 로그인

팔로워 180,962명


  • Source: Unknown