https://www.byanshuman.page/Writing on systems, GPUs, and distributed runtimes.enhttps://www.byanshuman.page/writing/three-hundred-processes-waited-for-one-pidhttps://byanshuman.page/writing/three-hundred-processes-waited-for-one-pidFri, 31 Jul 2026 00:00:00 GMTThe first description of HAMi issue 1662 blamed a file called /tmp/vgpulock/lock. Two hundred to three hundred processes were starting at once. Container logs repeated unified_lock locked, waiting 1 second, and a CUDA initialization that should have been quick took roughly a minute.hamicudakuberneteslinuxconcurrencyhttps://www.byanshuman.page/writing/the-tensor-was-only-the-handlehttps://byanshuman.page/writing/the-tensor-was-only-the-handleWed, 29 Jul 2026 00:00:00 GMTI began with a question that seemed too small for an article about an entire framework: when I write x, what exactly do I have? I had written reshape, not clone, and still acquired new storage. The transpose itself moved no values. It changed metadata from row-major strides (4, 1) to (1, 4). Flattening that order…pytorchsystemspythonml-systemshttps://www.byanshuman.page/writing/one-collective-call-used-two-generationshttps://byanshuman.page/writing/one-collective-call-used-two-generationsTue, 28 Jul 2026 00:00:00 GMTThe first two-machine run printed nothing. Locality zero contributes 1. Locality one contributes 2. Both should get 3. The same source worked with one locality, where it triumphantly proved that (1 = 1), but the two-locality process sat there until it was interrupted.hpxcollectivescppdistributed-systemshttps://www.byanshuman.page/writing/arrays-become-matriceshttps://byanshuman.page/writing/arrays-become-matricesMon, 27 Jul 2026 00:00:00 GMTThe vector owns one contiguous run of storage, a fact established when a retained pointer watched a vector grow in the address investigation. Nothing about those six integers says matrix. C++ sees a vector with six elements.cpplinear-algebramemoryperformancehttps://www.byanshuman.page/writing/my-canary-got-biggerhttps://byanshuman.page/writing/my-canary-got-biggerSun, 26 Jul 2026 00:00:00 GMTThe previous chapter reruns today’s two-configuration research scaffold. This final investigation preserves an earlier end-to-end session at CommCanary commit 02d894a, where the CLI reducer’s default oracle used five configurations. That is why the later ledger says six oracle calls and forty ranking relations…ml-systemsncclbenchmarkingdistributed-systemshttps://www.byanshuman.page/writing/one-hundred-events-became-onehttps://byanshuman.page/writing/one-hundred-events-became-oneSun, 26 Jul 2026 00:00:00 GMTA one hundred event communication trace went into a reducer. One event came out. The oracle accepted it. The reducer is commcanary reduce, a ddmin-style minimizer. Delta debugging is the old, honorable idea: delete chunks of the input, keep any deletion that preserves a chosen property, repeat until nothing more can…ml-systemsncclbenchmarkingdistributed-systemshttps://www.byanshuman.page/writing/the-address-where-the-value-livedhttps://byanshuman.page/writing/the-address-where-the-value-livedWed, 08 Jul 2026 00:00:00 GMTvalues[0] was still 17. The pointer to values[0] was already dead. An ordinary optimized build printed 17 twice on one run. That was the least helpful result it could have produced. The last line looked correct, so the program looked correct.cppmemoryallocatorsperformancehttps://www.byanshuman.page/writing/the-head-pointer-came-backhttps://byanshuman.page/writing/the-head-pointer-came-backTue, 07 Jul 2026 00:00:00 GMTThe compare-and-swap succeeded. Every node it had observed had already left the queue. The delayed operation read head pointer D, then stopped. Another operation removed D. The storage pool reused that exact address for a later node. Enough queue operations ran for the head pointer to contain D again.cppconcurrencyatomicsperformancehttps://www.byanshuman.page/writing/the-syscall-the-kernel-refusedhttps://byanshuman.page/writing/the-syscall-the-kernel-refusedTue, 07 Jul 2026 00:00:00 GMTThe first call returned a file descriptor. The second returned Operation not permitted. The path did not change. The bytes in the file did not change. The process kept the same user ID. Its working directory stayed where it was. The permission bits were still 0600, which allowed the owner to read and write. One…securitysyscallssandboxmacoshttps://www.byanshuman.page/writing/idle-gaps-are-not-computehttps://byanshuman.page/writing/idle-gaps-are-not-computeMon, 06 Jul 2026 00:00:00 GMTThe collectives chapter ended with algorithms whose cost changes with message size, topology, and arrival skew. A production tuner has to choose among those paths using incomplete evidence. One NCCL configuration looked ordinary until the workload ran.ml-systemsncclgpubenchmarkinghttps://www.byanshuman.page/writing/rings-trees-and-butterflies-hpx-collectives-dissectedhttps://byanshuman.page/writing/rings-trees-and-butterflies-hpx-collectives-dissectedMon, 06 Jul 2026 00:00:00 GMTEvery machine calls it, every machine gets back the same total, and nobody wrote a single send or recv. This is a collective: an operation that a whole group of machines performs together, the primitive underneath MPI_Allreduce and a gradient reduction in a distributed training run. The GPU investigation followed…hpxcollectivesdistributedhttps://www.byanshuman.page/writing/the-page-that-was-not-therehttps://byanshuman.page/writing/the-page-that-was-not-thereMon, 06 Jul 2026 00:00:00 GMTmmap returned a pointer. None of the memory behind it was resident. The mapping covered 4 MiB. The first loop loaded one byte from every 16 KiB page. It caused exactly 256 minor faults, and all 256 pages became resident. The second loop executed the same source loads without another page fault.osmemoryperformancearchitecturehttps://www.byanshuman.page/writing/how-your-computer-fakes-randomnesshttps://byanshuman.page/writing/how-your-computer-fakes-randomnessSun, 05 Jul 2026 00:00:00 GMTThe program never called srand. Nothing in its source selected 16807, yet the first process and the third process agreed down to the last bit. A process restart had apparently erased every variable in the program and restored one hidden variable to the same value.probabilitystatisticsprnghttps://www.byanshuman.page/writing/one-line-to-machine-codehttps://byanshuman.page/writing/one-line-to-machine-codeTue, 23 Jun 2026 00:00:00 GMTFour details did not fit the simple word “compile.” The matrix multiplication became extern_kernels.mm, which is a call through an Inductor lowering rather than a loop visible in this wrapper. The transpose became a different set of strides passed to reinterpret_tensor; no copy appears. The add and relu became…compilerspytorchml-systemshttps://www.byanshuman.page/writing/teaching-a-modelhttps://byanshuman.page/writing/teaching-a-modelMon, 15 Jun 2026 00:00:00 GMTThe loss reached zero while the big matrix stayed untouched. That sounds like the patch learned the task. It did learn the rows I gave it. Later, the same experiment made the model substantially worse on unrelated text. A second patch reached the same training accuracy and disagreed on a held-out prompt. A third…fine-tuningloraalignmenthttps://www.byanshuman.page/writing/one-gradient-backwardhttps://byanshuman.page/writing/one-gradient-backwardWed, 03 Jun 2026 00:00:00 GMTThe tape and PyTorch print the same ten decimal places. The central difference with h=0.1 differs by 0.0008778921. Calling the third row wrong would already assume the first two are the oracle. It is more useful to ask what each method computed. The tape applies stored local derivative rules. PyTorch executes its own…autogradtrainingdeep-learninghttps://www.byanshuman.page/writing/one-forward-passhttps://byanshuman.page/writing/one-forward-passWed, 27 May 2026 00:00:00 GMTParis is fifth. GPT-2 assigns 8.5 percent to ' the' and 3.2 percent to ' Paris'. The output is not one selected answer. It is a distribution across 50,257 possible next tokens, and the table shows its five largest entries.transformersinferencequantizationhttps://www.byanshuman.page/writing/how-machines-agreehttps://byanshuman.page/writing/how-machines-agreeFri, 22 May 2026 00:00:00 GMTA says 5. B says 4. The client sent five increments and there’s no honest answer to “what is the count,” because the two machines that are supposed to be the same machine now disagree, and nothing in this design will ever fix it. There’s no line of code that reconciles them, no moment where B notices it’s behind. The…distributed-systemsconsensusnetworkinghttps://www.byanshuman.page/writing/two-machines-talking-to-each-otherhttps://byanshuman.page/writing/two-machines-talking-to-each-otherMon, 04 May 2026 00:00:00 GMTCopper. Glass. Sometimes radio. That is the entire physical inventory of networking. Everything else, Ethernet, IP, TCP, RDMA, NCCL, the whole 600-page tower of acronyms, is a way of dressing those three media so that a 1 you put in over here comes out as a 1 over there, with some guarantees about how often it doesn’t.ethernethttps://www.byanshuman.page/writing/where-the-time-goeshttps://byanshuman.page/writing/where-the-time-goesThu, 30 Apr 2026 00:00:00 GMTHere’s a program that does exactly one interesting thing. It opens a file and adds up every byte. Twenty-one seconds to add up 67 million numbers. My phone can add 67 million numbers in a blink. And look at the split: 4.61 user is time my code spent running, 15.59 sys is time spent inside the kernel. Three-quarters…performanceossystemshttps://www.byanshuman.page/writing/a-file-is-just-byteshttps://byanshuman.page/writing/a-file-is-just-bytesFri, 24 Apr 2026 00:00:00 GMTOne column came back in 3 milliseconds from one file and 114 milliseconds from the other. Both files had two million logical records with fields named id, category, price, and qty. That is not quite the same as saying they contained the same information. The CSV carried characters and separators but no authoritative…dataparquetstoragehttps://www.byanshuman.page/writing/the-fast-path-out-of-pythonhttps://byanshuman.page/writing/the-fast-path-out-of-pythonMon, 20 Apr 2026 00:00:00 GMTThe same ten million input values, and numpy is eighty-five times faster. I first called them the same additions and the answer bit-for-bit identical. Both statements survive only with qualifications that became important later. Numpy changes the grouping of the additions, and this particular input happens to sum…pythonnumpyperformancehttps://www.byanshuman.page/writing/two-threads-to-a-task-runtimehttps://byanshuman.page/writing/two-threads-to-a-task-runtimeWed, 15 Apr 2026 00:00:00 GMTThere is no typo; the program says what its author meant. And yet a million increments evaporated. This is the first unpleasant fact of concurrency: correct-looking code can produce a different wrong answer every run.concurrencyatomicssystemshttps://www.byanshuman.page/writing/from-a-c-loop-to-a-gpu-kernelhttps://byanshuman.page/writing/from-a-c-loop-to-a-gpu-kernelSat, 04 Apr 2026 00:00:00 GMTThe machine refused to give me that exact number. (The 460x came from cross-build arithmetic: the unoptimized naive time divided by an optimized build’s Accelerate sample, 385.4 ms over ~0.84 ms. Two numbers that never lived in the same binary.)gpumetalcpphttps://www.byanshuman.page/writing/sixteen-million-and-onehttps://byanshuman.page/writing/sixteen-million-and-oneThu, 26 Mar 2026 00:00:00 GMTNot 16777217.0. The number sixteen million seven hundred seventy-two thousand two hundred sixteen survives the addition. The one disappears. Why is 16777216.0f + 1.0f still 16777216.0f? And why does the same shape of thing happen when summing gradients, computing variance, doing a dot product, reducing across…numericsfloating-pointcpphttps://www.byanshuman.page/writing/from-source-to-a-running-processhttps://byanshuman.page/writing/from-source-to-a-running-processSat, 21 Mar 2026 00:00:00 GMTThe first program was ten lines long. It uses the templates and value categories from the previous article, but understanding the template is not the puzzle here. I wanted to know what the machine received after C++ had finished with it.cpptoolchainlinkershttps://www.byanshuman.page/writing/python-is-a-c-programhttps://byanshuman.page/writing/python-is-a-c-programSun, 15 Mar 2026 00:00:00 GMTThe first weird thing was the size. On this Mac, the python3.13 executable is not a giant interpreter binary. It is 52,640 bytes. Fifty-two kilobytes is not enough room for the whole Python runtime, the parser, the compiler, the object system, dictionaries, lists, integers of unbounded size, exceptions, imports,…pythoncpythonchttps://www.byanshuman.page/writing/std-move-doesnt-move-anythinghttps://byanshuman.page/writing/std-move-doesnt-move-anythingWed, 11 Mar 2026 00:00:00 GMTI finally opened the file where std::move lives on my Mac: /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__utility/move.h. I had been typing the name for weeks because the HPX code I was studying was full of it. Somewhere along the way I had started treating it like a compiler instruction…cppmove-semanticshpx