I came to a similar idea of professor blamocur, but a little different.
[imath]V_{\text{exc}}[/imath] is the excluded volume between two molecules. I will include a picture to illustrate that volume.
View attachment 38110
Vector [imath]\displaystyle \bold{r_{12}}[/imath] goes from the center [imath]A[/imath] of the first molecule to the center [imath]B[/imath] of the second molecule (or reverse). The volume differential element [imath]d\bold{r_{12}}[/imath] can be treated like [imath]d\bold{V}[/imath]. It is complicated to illustrate this, but [imath]d\bold{r_{12}}[/imath] must be a very small volume of that excluded volume, its direction is going from the center [imath]B[/imath] toward the center [imath]A[/imath] (or from the circumeference of [imath]A[/imath] molecule toward the center [imath]B[/imath] horizontally).
If we think of the volume differential element [imath]d\bold{r_{12}}[/imath] like [imath]d\bold{V}[/imath], then a very small volume of a sphere is equal to [imath]4\pi r^2 \ dr[/imath], but we know the radius of our sphere is [imath]|\bold{r_{12}}| = r_{12}[/imath], then we have:
[imath]d\bold{r_{12}} = 4\pi r^2_{12} \ dr_{12}[/imath]
But [imath]d\bold{r_{12}}[/imath] is a vector, so we must give a direction to it, then we have:
[imath]d\bold{r_{12}} = 4\pi r^2_{12} \ dr_{12} \ d\hat{r}_{12}[/imath]
The unit vector [imath]d\hat{r}_{12}[/imath] means that the direction of this differential element is along the centers [imath]A[/imath] and [imath]B[/imath] of the two molecules.
And since I took an arbitrary differential element [imath]d\bold{V}[/imath], I included the [imath]4\pi[/imath]. The book must has worked in a specific coordinate system, which not very obvious to me, so they should get the [imath]4\pi[/imath] during integration.
My final guess is that, the volume differential element is:
[imath]d\bold{r_{12}} = r^2_{12} \ dr_{12} \ d\hat{r}_{12}[/imath], and the [imath] 4\pi[/imath] will eventually appear from calculating the extra stuff inside [imath]V_{\text{exc}}[/imath].