nice work btw

The 6 wire interface between the AVR I/O pins and the display is very clean.
Unlike the quirky interfaces with obscure exceptions that I’m often forced to use.

Because I am too often pin-limited in my systems, I would be tempted to use fewer pins:

* could eliminate the “row” pin by making the one-hot row select 74HC595 automatically select the next row whenever you finish loading data into the column bits, by wiring its SCK to the global RCK. But once the one-hot has been shifted out the end, it gets a bit tricky resetting it to the beginning — probably requiring more hardware.

* could use a 3 pin “daisy-chain spi”, with “data”, “SCK”, and “RCK” pins, by connecting the “data-in” of each shift register to the cascading “serial data out” of the previous 74HC595 in the daisy chain. But your method loads the 4 chips in 1/3 or 1/4 the time.

Because I like nice, straight, parallel lines rather than many-to-many crossing permutations,
I would be tempted to connect the LEDs as a “packed pixel” array.
The “packed pixel” array would connect the first 3 columns r,g,b to the first 3 pins of the first 74HC595 shift register, the second 3 columns r,g,b to the next 3 pins of that first shift register, and the third 3 columns r,g,b, to the last 2 pins of that first shift register and the first pin of the next register … but splitting that pixel across 2 chips probably makes the software more complex than your “bit plane” approach.

Years ago, some people would “brightness sort” their LEDs, putting the brightest LEDs at the top of the array and the dimmest LEDs at the bottom of the array.
Is that really necessary with modern LEDs?