Split learning at equal bits

This entry is not about language models. It is a split-learning test on CIFAR-10 images. A client runs the frozen first two stages of a small image network, and then sends one short message for each image to a server. Most other entries of the lab test latent messages between language models. This entry tests the same idea outside them: a symbolic message as the control for a learned message with an equal number of bits.

The test measures two values at the same cap of bits: test accuracy, and leakage. Leakage is the PSNR of the image that a trained decoder rebuilds from the message; a higher value means that more of the input leaks. The symbolic message TOPK@64 sends the first class and the gaps to the other nine class scores of a small client head, in 58 bits. The learned bottleneck BOT@64 sends 16 channels on a 2×2 grid at 1 bit each, in 64 bits.

The sealed study used 3 seeds, 10,000 test images and the smaller decoder (Decoder-S). TOPK@64 kept 13.67 points more test accuracy than BOT@64 (95% CI 13.09 to 14.27; 0.826 against 0.690). BOT@64 leaked 0.550 dB more (CI 0.527 to 0.572). With these two outcomes, the preregistered rule gives case S, “symbolic dominates”. The leakage margin was thin: the rule needs a mean of at least 0.5 dB, and seed 2 alone gave 0.489 dB. Also, the CI came from a bootstrap over images only, without the variance between seeds.

A preregistered replication then tested the leakage result again. It used 6 new seeds, the larger decoder (Decoder-L) as the primary decoder, and a two-level bootstrap over seeds, then images. The leakage gap was 0.609 dB (CI 0.547 to 0.664), reading R1, “L+ replicates”. One seed of six gave 0.480 dB, below the margin. Decoder-S gave 0.615 dB, and the accuracy gap was 13.85 points (CI 13.15 to 14.57). With the seed variance, the CI is 0.117 dB wide, against 0.045 dB from images only.

In the secondary rows, TOPK was at least as accurate as the activation messages at each budget. At 1,024 bits, the bottleneck had the accuracy of TOPK (148 bits) and leaked 3.8 dB more. At 64 bits, the label alone (4 bits) had the accuracy of TOPK and 0.33 dB less leakage. Its NLL was worse by 0.26 nats. Two independent scripts recomputed the values of the two studies from the saved files, and both audits passed.

Limits: the leakage values are lower bounds, from one decoder family that learns from 20,000 pairs, and stronger attackers exist. The bottleneck was much weaker than the design expected, so the pair compares a strong symbolic message with a weak learned message. The study has one data set, one model family, a frozen client, forward messages only and a pixel metric; the replication changed only the seeds. A symbolic message is not private: TOPK@64 still leaks 0.36 dB above the floor of the true label. Cut C1 and an end-to-end trained client did not run.