Yes image directive can be usefuill, however consider that actually image size is acceptable for html out but not for pdf, because of this I suggest a way to control the includegraphic statement.
I'm already using pdf output.
I gave a try also to defaultFontName but as result I got a graph almost the same size but with font very small compared with actor dimensions.
BTW: sphinx with plantUML is the tool I have looked for along many years!!!
I forgot to sign the preceeding post. 8-)
My 2 cents follows.
This way you can use the directive
:includegraphics: scale=1, width=... etc