I started reading about parallel make since I have a build process that takes quite a while. What a pain, I quickly (within a minute) discarded the idea as a waste of time, and figured I could do it in bash.

The main problem is that I have a slow script (that I can’t be bothered rewriting) that is run over and over again to generate different parts of my website.

So the target used to look like this:

shapehtml-target: force
        ./genshapehtml 96 58 4 arrowleft 0 0 18
        ./genshapehtml 96 58 4 arrowright 0 0 18
        ./genshapehtml 96 39 4 capsule 0 0 18
        ./genshapehtml 156 39 4 capsule-wide 0 0 18
        ./genshapehtml 96 58 4 fun 0 0 18
        ./genshapehtml 103 93 4 heart 0 -7 18
        ./genshapehtml 96 96 4 octagon 0 0 18
        ./genshapehtml 96 58 4 oval 0 0 18
        ./genshapehtml 96 57 4 rounded 0 0 18
        ./genshapehtml 156 57 4 rounded-wide 0 0 24
        ./genshapehtml 104 58 4 shard 0 -2 18
        ./genshapehtml 102 41 4 trapezoid 0 0 18

So when I time this part it takes 2m 23 seconds, a complete eternity.

[cameron@mars buttons]$ time make shapehtml-target
./genshapehtml 96 58 4 arrowleft 0 0 18
...
./genshapehtml 102 41 4 trapezoid 0 0 18

real    2m23.236s
user    0m44.110s
sys     1m43.508s

but I just run all the scripts in the background, and added a wait at the end. So now the target looks like this:

shapehtml-target: force
        ./genshapehtml 96 58 4 arrowleft 0 0 18 &\
        ./genshapehtml 96 58 4 arrowright 0 0 18 &\
        ./genshapehtml 96 39 4 capsule 0 0 18 &\
        ./genshapehtml 156 39 4 capsule-wide 0 0 18 &\
        ./genshapehtml 96 58 4 fun 0 0 18 &\
        ./genshapehtml 103 93 4 heart 0 -7 18 &\
        ./genshapehtml 96 96 4 octagon 0 0 18 &\
        ./genshapehtml 96 58 4 oval 0 0 18 &\
        ./genshapehtml 96 57 4 rounded 0 0 18 &\
        ./genshapehtml 156 57 4 rounded-wide 0 0 24 &\
        ./genshapehtml 104 58 4 shard 0 -2 18 &\
        ./genshapehtml 102 41 4 trapezoid 0 0 18 &\
        wait

and the result of running this parallel target: 17 seconds. Quite acceptable. (around 1/10th of the time)

[cameron@mars buttons]$ time make shapehtml-target
./genshapehtml 96 58 4 arrowleft 0 0 18 &\
 ./genshapehtml 96 58 4 arrowright 0 0 18 &\
...
./genshapehtml 102 41 4 trapezoid 0 0 18 &\
 wait

real    0m17.413s
user    0m35.692s
sys     1m32.897s
[cameron@mars buttons]$

Of course, your mileage might vary, I have I7 quad core with hyperthreading and a relatively fast SSD. And while the script is running, you notice much better CPU ultization.

If you notice the CPU is pegged, then just don’t run as many in parallel, and add wait after a couple. eg:

shapehtml-target: force
        ./genshapehtml 96 58 4 arrowleft 0 0 18 &\
        ./genshapehtml 96 58 4 arrowright 0 0 18 &\
        ./genshapehtml 96 39 4 capsule 0 0 18 &\
        ./genshapehtml 156 39 4 capsule-wide 0 0 18 &\
        wait
        ./genshapehtml 96 58 4 fun 0 0 18 &\
        ./genshapehtml 103 93 4 heart 0 -7 18 &\
        ./genshapehtml 96 96 4 octagon 0 0 18 &\
        ./genshapehtml 96 58 4 oval 0 0 18 &\
        wait
        ./genshapehtml 96 57 4 rounded 0 0 18 &\
        ./genshapehtml 156 57 4 rounded-wide 0 0 24 &\
        ./genshapehtml 104 58 4 shard 0 -2 18 &\
        ./genshapehtml 102 41 4 trapezoid 0 0 18 &\
        wait

And similarly you might want to check the hard drive usage to make sure each process is not contending with each other.

The big drawback here is that if genshapehtml had failed with an exit code, the make will continue on.  In my case, this never happens (I always exit 0), but I could add a command after the wait to ensure that it succeeded: eg each program  writes success/fail to a file, and the testprogram checks that they all succeeded.

Leave a Reply