background

Niko's Project Corner

Introduction to Stable Diffusion's parameters
Description Learning to use a pre-trained SD model.
Languages Python
PyTorch
Tags Com­puter Vi­sion
Au­toen­coder
Sta­ble Dif­fu­sion
Duration Fall 2022
Modified 10th November 2022
thumbnail
Sta­ble Dif­fu­sion is an im­age gen­er­ation net­work, which was re­leased to the pub­lic in 2022. It is based on a dif­fu­sion pro­cess, in which the model gets a noisy im­age as an in­put and it tries to gen­er­ate a noise-free im­age as an out­put. This pro­cess can be guided by de­scrib­ing the tar­get im­age in plain En­glish (aka txt2img), and op­tion­ally even giv­ing it a tar­get im­age (aka. img2img). This ar­ti­cle doesn't de­scribe how the model works and how to run it your­self, in­stead this is more of a tu­to­rial on how var­ious pa­ram­eters af­fect the re­sult­ing im­age. Non-tech­ni­cal peo­ple can use these im­age gen­er­at­ing AIs via web­pages such as Ar­tis­tic.wtf (my and my friend's pro­ject), Craiyon.com, Mid­jour­ney.com and others.

This ar­ti­cle shows how three dif­fer­ent pa­ram­eters (num­ber of steps, "CFG" or clas­si­fier-free guid­ance scale, and img2img strength) af­fect the im­age. These may in­ter­act with each other in com­plex ways, but that isn't stud­ied here. Ex­per­iments use five dif­fer­ent in­put texts (aka "prompts"): "a white par­rot sit­ting by the sea", "full length por­trait of gor­geous god­dess stand­ing in field full of flow­ers", "panda movie poster with yel­low back­ground and red ti­tle text", "moun­tains by a river" and "an an­cient yel­low robot hold­ing a sword on de­sert". In ad­di­tion they con­tain "prompt-en­gi­neer­ing" terms such as "cin­ematic, highly de­tailed, il­lus­tra­tion, con­cept ar". These greatly im­prove the re­sults, but that topic isn't cov­ered here ei­ther. One can find prompt ex­am­ples from sites such as lex­ica.art and hug­ging­face.co's Mag­icPrompt.

Vari­ations of the "same" im­age are gen­er­ated by chang­ing the seed of a ran­dom num­ber gen­er­ator. This af­fects the noise from which de­nois­ing steps are started (the dif­fu­sion pro­cess), and the al­go­rithm will con­verge to quite dif­fer­ent look­ing im­ages. But they should still be con­sep­tu­ally sim­ilar. Here three dif­fer­ent seeds are used for each ex­per­iment.

Each pa­ram­eter is var­ied to eight dif­fer­ent val­ues, ei­ther on a lin­ear or log­arith­mic scale. Thus each fig­ure con­sists of 3 × 8 sub-im­ages. Orig­inally each im­age was gen­er­ated on a res­olu­tion of 512 × 512 (ex­cept the panda poster, since it has a dif­fer­ent as­pect ra­tio), so the re­sult­ing im­age tile would have a res­olu­tion of 1536 × 4096, and there are 20 such im­ages in to­tal. This brings the to­tal im­age size to 156 megapix­els, which would make the file size rel­atively large. Orig­inal PNGs take 180 MB, but the used im­ages here have a res­olu­tion of 2560 × 960, and with a JPG qual­ity of 90% their to­tal size is only 12 MB (93% less).

steps_a_white_parrot_sitting_by_the_se_seagull_1
Figure 1: "a white par­rot sit­ting by the sea", gen­er­ated at vary­ing num­ber of de­nois­ing steps. Ma­jor de­tails have con­verged by the 4th or 5th column (15 & 18 steps), but there is still a no­tice­able jump on the last two columns (21 & 50 steps). The "sea" back­ground lacks any in­ter­est­ing de­tails.

The de­fault set­tings are 50 de­nois­ing steps, CFG scale of 7.5 and ref­er­ence im­age strength of 0 (mean­ing no ref­er­ence im­age). All other pa­ram­eters are kept con­stant while one other pa­ram­eter is var­ied. (Note: the com­mmon con­ven­tion on img2img ap­pli­ca­tions seems to spec­ify the weight of the added ran­dom noise, not the weight of the in­put im­age! Here the "im­age strength" is 1 - "noise strength".) The re­sults shown here are based on Sta­ble Dif­fu­sion model ver­sion 1.4 and using the "DDIM" sampler.

The first var­ied pa­ram­eter is the num­ber of de­nois­ing steps. The used val­ues are [3, 6, 9, 12, 15, 18, 21, 50], and usu­ally the im­age has con­verged by step 50. These re­sults are shown in fig­ures 1 - 5.

steps_an_ancient_yellow_robot_holding_guy_1
Figure 2: "an an­cient yel­low robot hold­ing a sword on de­sert" at vary­ing num­ber of steps. The re­sult is quite ac­cept­able al­ready by column four (12 steps), and by 24 steps it has con­verged so much that the re­sult is very sim­ilar as when us­ing 50 steps (the right-most column).
steps_hyperrealistic_full_length_portr_korean_woman_1
Figure 3: "full length por­trait of gor­geous god­dess stand­ing in field full of flow­ers" at vary­ing num­ber of steps. Here the con­ver­gence is quite slow, and the pose keeps chang­ing through­out the pro­cess. De­pend­ing on the seed, steps 24 and 50 may pro­duce very dif­fer­ent look­ing im­ages. This means that us­ing a lower step-count for pre­view­ing seeds doesn't al­ways work.

All im­ages have their ma­jor as­pects de­fined al­ready with just 10 - 15 de­nois­ing steps. Most of the time even go­ing from 24 to 50 steps doesn't change the re­sult in any ma­jor way, but of course there are ex­cep­tions. The most chal­leng­ing in­puts were "full length por­trait of gor­geous god­dess stand­ing in field full of flow­ers" (Fig­ure 3 and "panda movie poster with yel­low back­ground and red ti­tle text" (Fig­ure 5). They don't have much in com­mon, but the "god­dess" has a com­plex fore­ground and back­ground, and the "panda" has ty­pog­ra­phy (seems to have ei­ther Chi­nese or En­glish de­pend­ing on the seed), high-con­trast­ing col­ors and just a vague idea on how the com­po­si­tion should look like.

This is in­con­trast to the "a white par­rot sit­ting by the sea" (Fig­ure 1) and "an an­cient yel­low robot hold­ing a sword on de­sert" (Fig­ure 2), which have a some­what con­strained pose and a sim­pler back­ground. Al­though the set of all pos­si­ble robots has huge va­ri­ety.

steps_mountains_by_a_river_fantastic_l_mountains
Figure 4: "moun­tains by a river" at vary­ing num­ber of steps. Ini­tial im­ages lack con­trast and de­tail, but again the column 4 (12 steps) re­sem­bles the fi­nal re­sult very much. Some­how the art style seems foggy and lacks de­tail, nat­urally this can be guided by chang­ing the prompt by in­clud­ing terms like "sun­shine".
steps_panda_movie_poster_with_yellow_b_kungfu_panda
Figure 5: "panda movie poster with yel­low back­ground and red ti­tle text" at vary­ing num­ber of steps. Even the column three (9 steps) could pass as a low-qual­ity poster, but the con­tents con­tinue to change un­til ap­prox­imately column 6 (18 steps). And es­pe­cially the top right cor­ner's im­age ben­efits from "full" 50 steps, which adds de­tail to its hair and tunes the pose to look straight at the cam­era. It is plau­si­ble that orig­inal Kung Fu Panda movie posters were part of the train­ing dataset.
Step pa­ram­eter sum­mary: This pa­ram­eter is an easy one to tune, since the end re­sult is al­most al­ways bet­ter with more steps. Some­times the end re­sult has con­verged af­ter 30 steps, but it is bet­ter to let it run for 50 steps to be sure. And it is very rare that the end re­sult would sig­nif­icantly change when go­ing be­yond that, at least when gen­er­at­ing 512 × 512 res­olu­tion im­ages.

The next stud­ied pa­ram­eter is the "CFG" or clas­si­fier-free guid­ance scale, or just "scale". It is de­scribed for ex­am­ple at dif­fu­sion-news.org, but the con­clu­sions there are con­flict­ing with ex­per­imen­tal re­sults which are shown in this ar­ti­cle. Ac­cord­ing to the ar­ti­cle, it "is a mea­sure of how close you want the model to stick to your prompt when look­ing for a re­lated im­age to show you. A Cfg Scale value of 0 will give you es­sen­tially a ran­dom im­age based on the seed, where as a Cfg Scale of 20 (the max­imum on SD) will give you the clos­est match to your prompt that the model can pro­duce." They even run the tests with var­ious dif­fer­ent sam­plers, but those aren't cov­ered in this ar­ti­cle for the sake of stick­ing to ba­sics.

Tested CFG scales are from the for­mula 1 + 6.5 * 10i, where i is in­ter­po­lated lin­early be­tween -1 and 1. This was cho­sen so that the de­fault scale of 7.5 is in the mid­dle. The val­ues (rounded to two dec­imals) are [1.65, 2.25, 3.42, 5.68, 10.03, 18.44, 34.67, 66.0]. But the ac­tual mid­dle value 7.5 isn't pre­sent in the list, since there is an even num­ber of sam­ples. Ex­am­ples are shown in fig­ures 6 - 10.

scales_a_white_parrot_sitting_by_the_se_seagull_1
Figure 6: "Par­rot" at vary­ing CFG scale be­tween 1.65 and 66.0. Lower scale val­ues pro­duce an in­ter­est­ing hand-drawn style, but on the larger scales the re­sults aren't very suc­cess­ful.
scales_an_ancient_yellow_robot_holding_guy_1
Figure 7: "Robot" at vary­ing CFG scale be­tween 1.65 and 66.0. Small­est val­ues aren't re­ally co­her­ent, but larger val­ues pro­duce an in­ter­est­ing pop-art style.

The scale pa­ram­eter has a very con­crete im­pact on the re­sult­ing artis­tic style. Smaller num­bers seem to cause the it­er­ation to fo­cus on smaller scale de­tail, and the over­all con­trast and sat­ura­tion is fairly weak. But if such artis­tic style is de­sired, con­trast and sat­ura­tion can be eas­ily fixed in post-pro­cess­ing. Pa­ram­eters 6 - 8 give the most re­al­is­tic and well-bal­anced re­sults. Go­ing be­yond that, the re­sults get pro­gres­sively to the op­po­site di­rec­tion. They have very vi­brant col­ors, have high global con­trast but are lack­ing in small de­tails.

CFG scale pa­ram­eter sum­mary: Use smaller val­ues for in­tri­cate hand-drawn style, the de­fault 7.5 for re­al­is­tic pho­tos and larger val­ues for pop-art style with high con­trast and sat­urated col­ors. But go­ing to ei­ther ex­treme may re­quire fur­ther tweak­ing in a photo-edit­ing soft­ware.
scales_hyperrealistic_full_length_portr_korean_woman_1
Figure 8: "God­des" at vary­ing CFG scale be­tween 1.65 and 66.0. Small­est scales are well suited for this prompt, but larger val­ues don't work that well.
scales_mountains_by_a_river_fantastic_l_mountains
Figure 9: "Moun­tains" at vary­ing CFG scale be­tween 1.65 and 66.0. Here all scales pro­duce quite de­cent but very dif­fer­ent re­sults.
scales_panda_movie_poster_with_yellow_b_kungfu_panda
Figure 10: "Panda" at vary­ing CFG scale be­tween 1.65 and 66.0. Here smaller val­ues pro­duce in­ter­est­ing com­po­si­tions with lots of de­tail, and larger val­ues pro­duce less in­ter­est­ing im­ages.

The next stud­ied pa­ram­eter is the img2img strength (or "weight"), a value be­tween 0% and 100%. The all pre­vi­ously de­scribed pa­ram­eters still play a role, but here they are kept to their de­fault val­ues (run­ning for 50 steps and the CFG scale is 7.5). Note that be­cause of rea­sons, the strength has an op­po­site con­ven­tion that orig­inal work such as source code at github.com/Com­pVis/sta­ble-dif­fu­sion which says "strength is a value be­tween 0.0 and 1.0, that con­trols the amount of noise that is added to the in­put im­age. Val­ues that ap­proach 1.0 al­low for lots of vari­ations but will also pro­duce im­ages that are not se­man­ti­cally con­sis­tent with the in­put." Ba­si­cally it is a trade-off pa­ram­eter be­tween the im­age gen­er­ated purely from the prompt, and the tar­get im­age. 0% noise means that the out­put re­sem­bles the tar­get 100%, and 100% noise means that the tar­get im­age has no im­pact. Hope­fully this isn't con­fus­ing any­body.

The weight is a very del­icate pa­ram­eter to tune, and if the two im­ages (from the prompt and the tar­get) are very dif­fer­ent, then there might be a tip­ping point where chang­ing the weight even a lit­tle bit will have a large change in the re­sult­ing im­age. How­ever this de­pends very much on the con­text, and on many ex­per­iments the re­sults had a nice gradu­lar change. But be­cause of this, the img2img ex­eper­iments were run with two dif­fer­ent pa­ram­eter gra­di­ents. The first one uses weights [0.03, 0.06, 0.09, 0.12, 0.15, 0.18, 0.21], and the sec­ond one uses [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7]. The re­sults are shwon in fig­ures 11 - 20. Note that only seven dif­fer­ent weights were used, but im­ages con­sist of eight columns. This is be­cause the right-most pic­ture al­ways shows the tar­get im­age (cor­re­spond­ing to a strength of 1.0, or noise-level of zero).

img2img_a_white_parrot_sitting_by_the_se_seagull_1
Figure 11: "Par­rot" turn­ing into a seag­ull by the sea, us­ing small strength val­ues. All val­ues (be­tween 0.03 and 0.21) show in­ter­est­ing in­ter­po­la­tion be­tween the "source" and "tar­get" im­ages.
img2imgheavy_a_white_parrot_sitting_by_the_se_seagull_1
Figure 12: "Par­rot" turn­ing into a seag­ull by the sea, us­ing large strength val­ues. Columns 5 - 7 (strengths 0.5 - 0.7) re­sem­ble the tar­get im­age too much, and very lit­tle "par­rot" in­flu­ence is pre­sent.
img2img_an_ancient_yellow_robot_holding_guy_1
Figure 13: "Robot" on a de­sert, in­spired by a yel­low­ish stick-man draw­ing, us­ing small strength val­ues. In this case the tar­get im­age isn't re­al­is­tic at all, so at these strengths the im­age hasn't in­flu­enced the com­po­si­tion at all. It has only set the yel­low color theme.
img2imgheavy_an_ancient_yellow_robot_holding_guy_1
Figure 14: "Robot" on a de­sert, in­spired by a yel­low­ish stick-man drawin, us­ing large strength val­ues. Here strength val­ues of 0.3 - 0.5 seem to pro­duce the best re­sults, go­ing higher than this loses all de­tails since they are ab­sent from the tar­get im­age.
img2img_hyperrealistic_full_length_portr_korean_woman_1
Figure 15: "God­dess" mim­ick­ing a woman on a grass field, us­ing small strength val­ues. Here all im­ages are quite in­ter­est­ing, al­though the com­po­si­tion at the mid­dle column isn't ideal since the face isn't vis­ible.
img2imgheavy_hyperrealistic_full_length_portr_korean_woman_1
Figure 16: "God­dess" mim­ick­ing a woman on a grass field, us­ing large strength val­ues. Im­ages be­yond the 2nd column (hav­ing the strength of 0.2) aren't that good, fa­cial and other fea­tures are a mix-n-match with clearly vis­ible ar­ti­facts.
img2img_mountains_by_a_river_fantastic_l_mountains
Figure 17: "Moun­tains" tak­ing guidanse from a sim­ple draw­ing, us­ing small strength val­ues. Only the first three columns (strengths 0.03 - 0.09) show in­ter­est­ing re­sults. The taget im­age is so sim­plis­tic and un­re­al­is­tic that it doesn't have a good im­pact on the re­sult­ing im­age.
img2imgheavy_mountains_by_a_river_fantastic_l_mountains
Figure 18: "Moun­tains" tak­ing guidanse from a sim­ple draw­ing, us­ing large strength val­ues. Clearly large strengths don't work well in this case.

Fi­ugres 13, 14 ("robot"), 17 and 18 ("moun­tains") show that one must use very low strengths if the tar­get im­age is un­re­al­is­tic, hav­ing lots re­gions with uni­form color and lacks small-scale de­tails. Bet­ter re­sults could be ob­tained by run­ning the img2img gen­er­ation sev­eral times, us­ing a low strength value each time and pick­ing the most promis­ing look­ing im­age as a start­ing point for the next it­er­ation.

Other ex­am­ples show that best re­sults are ob­tained by us­ing a re­al­is­tic in­put im­age to be­gin with, if re­al­is­tic re­sults are de­sired. But since it is very con­text-de­pen­dent on which strength val­ues pro­duce res­on­able re­sults, trial-and-er­ror ap­proach seems the eas­iest one. And one must not for­get that CFG scale can be changed as well from the de­fault 7.5, and it will have a ma­jor im­pact on the out­come as well.

img2img_panda_movie_poster_with_yellow_b_kungfu_panda
Figure 19: "Panda" tak­ing guidanse from the ac­tual movie poster, us­ing small strength val­ues. Here all re­sults look more or less rea­son­able.
img2imgheavy_panda_movie_poster_with_yellow_b_kungfu_panda
Figure 20: "Panda" tak­ing guidanse from the ac­tual movie poster, us­ing large strength val­ues. By column 5 (strength 0.5) all of the re­sults look very sim­ilar. Most in­ter­est­ing re­sults are be­tween columns 2 and 4 (strengths 0.2 - 0.4).
Img2img strength pa­ram­eter sum­mary: The re­sults de­pend very much on the prompt and the tar­get im­age. Some cases work best with a very small weight, while oth­ers re­quire a larger one so trial-and-er­ror must be used. But in gen­eral smaller weights work best when the tar­get im­age is un­re­al­is­tic and lacks de­tail, and larger weights can be used when the tar­get im­age is aready re­al­is­tic.

There are still many more top­ics to study, such as what kind of de­scrip­tions to ap­pend to the prompt, us­ing dif­fer­ent ver­sions of the model (ver­sion 1.5 is al­reayd out), us­ing dif­fer­ent sam­plers among oth­ers.

Related blog posts:

VideoClustering
Agadmator
Bananagrams1
Puzzles
HelsinkiDeblur

Home Download all articles

Navigation

Home  (Home page)
About  (About me)
Platform  (About this blog)

External

LinkedIn  (Niko Nyrhilä)
GitHub  (nikonyrh)
Stackoverflow  (nikonyrh)

Most recent

Bruteforcing Countdown numbe...  (2023 Apr)
Cheating at Bananagrams with...  (2023 Apr)
Introduction to Stable Diffu...  (2022 Nov)
Matching puzzle pieces together  (2022 Jul)
Single channel speech / musi...  (2022 Feb)

Most frequent tags

Computer Vision  (13)
GitHub  (12)
Databases  (9)
Elasticsearch  (6)
FFT  (5)
Rendering  (5)
Applied mathematics  (4)

Most frequent languages

Python  (13)
C++  (11)
Matlab  (10)
Keras  (6)
Clojure  (6)
Bash  (6)
PHP  (6)

Co-occurrence matrix

Matl Pyth C++ Cloj Bash Kera
Comput 6 6 3 1 0 5
GitHub 0 2 1 4 3 0
Databa 0 3 2 2 1 0
Render 3 0 3 0 0 0
Nginx 0 1 0 0 4 0
Autoen 0 3 0 1 0 2
Elasti 0 2 0 3 0 0
FFT 3 1 1 0 0 1
Data S 2 1 2 1 0 1
JVM 0 1 0 3 1 0
Docker 0 1 0 0 3 0
FastCG 0 0 3 0 0 0
Applie 2 2 0 0 0 0
Field 2 0 2 0 0 0
Omnidi 2 0 2 0 0 0
Affine 2 0 2 0 0 0
Master 1 0 2 0 0 0
Archit 0 1 0 0 2 0
Visual 1 0 2 0 0 0
Spark 0 1 0 0 2 0
Blog 0 0 0 2 0 0
Hyphen 0 0 0 2 0 0
Stack 0 1 1 0 0 0
SQL 0 0 1 1 0 0
Busine 0 1 0 1 0 0
Signal 0 1 0 0 0 1
Encryp 0 0 0 0 1 0
Git 0 0 0 1 0 0
Stable 0 1 0 0 0 0
Redis 0 1 0 0 0 0
Thrust 0 0 1 0 0 0
Kibana 0 0 0 1 0 0
Astron 1 0 0 0 0 0
Mustac 0 0 1 0 0 0
NAT 0 0 0 0 1 0
jQuery 0 0 1 0 0 0
SSH 0 0 0 0 1 0
Happyh 0 0 1 0 0 0
Backup 0 0 0 0 1 0
Pthrea 0 0 1 0 0 0
AWS 0 0 0 0 1 0
SIFT 0 0 1 0 0 0
SURF 0 0 1 0 0 0
Conjug 0 0 1 0 0 0
Kalman 0 0 1 0 0 0
Partic 0 0 1 0 0 0
Gradie 0 0 1 0 0 0
Simult 0 0 1 0 0 0
Roboti 0 0 1 0 0 0
Princi 1 0 0 0 0 0
Receiv 1 0 0 0 0 0
Linear 1 0 0 0 0 0
Suppor 1 0 0 0 0 0
Machin 1 0 0 0 0 0
Discre 1 0 0 0 0 0