A practical guide to AI image generation, from diffusion to agents
Comments
vunderba
bananaflag
Here AGI means Adventure Game Interpreter.
satvikpendem
Exactly, turn something nondeterministic into something more deterministic and AI does a great job at that.
thijson
Looks great. This could be a form of image compression.
corysama
Several Amiga/DOS era games used a similar system for image compression.
armcat
This is very cool! I will try your method for background and tile creation in the asset-generation tool I've built [1]. Would your procedural method work for animations as well, like sprites?
vunderba
Hey armcat! I haven’t tried it extensively for generating sprite sheets, but here’s a quick attempt I made along with the prompt.
It works okay but you'd probably have to do a little bit of coaxing to get it to come out exactly right.
nonethewiser
I suspect your picscript language is an entirely unnecessary abstraction and the prompt (ie the model) is doing all the work.
badsectoracula
How so? Judging from the examples the model does seem to take advantage of the language (creates and uses reusable elements) and AFAICT the language itself compiles down to "real" Sierra PIC commands so in theory it can be used with a Sierra PIC viewer/engine.
nonethewiser
Yeah compiling to Sierra PIC is a unique advantage over asking the model for an image. But the claim im attacking is different. The claim in attacking is that you cant just tell it to produce images like this directly.
>As a result you end up sidestepping a lot of the problems with larger generative models, which can have issues with grid alignment, palette choices, and high-frequency noise.
All of these can be avoided with simple prompts and don’t require all this ceremony.
Convo then the artifact from the convo (sorry, apparently no way to share the pics but they are just captured from the artifact).
Convo: https://claude.ai/share/78cd63c8-ffdd-4569-8de2-ac8ab5c5cfe0
nsib96
the style is completly differrent…
nonethewiser
How so?
The difference i see is the shading. Could easily tell it to do flat fills instead.
Regardless it has none of the supposed problems of AI generated pixel art.
westsmith
Neat! I've followed you on Github. I like the idea of going to the past to see what works and adapting it for the LLM age. Nice work!
vunderba
Thanks! Yeah, I started building it because I read an interview with the original creators of King’s Quest, Ken and Roberta Williams where they mentioned in passing that it was too expensive (in the 80s) from a storage perspective to store entire bitmaps for each scene, so instead they'd just store a series of very simple drawing instructions and render on demand.
I’m super excited to see more LLM-driven procedural graphics stuff in the future!
fsloth
I think this is missing a nice hack.
Learn to draw.
Then ask the model to restyle the image.
You can get wonderful photographs with the exact composition and scene you want for all sorts of use cases.
Second hack.
Model the scene you want from low resolution geometry (boxes, spheres etc) for the composition. Then use that to ground the generated image (ie give it as reference) and ask the model to re-imagine it in the style you want.
micromacrofoot
You don't even need to be able to draw well, so learning basic composition may suffice
mikeaskew4
Boy are we all gonna be disappointed when it never learns to draw.
zer0tonin
Better hack: learn to draw. End of the story.
fsloth
Not really sure why removing the AI component would make anything better. They are a new modality for art and expression.
przmk
Are there people seriously thinking they’re making art by generating images with AI?
kouteiheika
> Are there people seriously thinking they’re making art by generating images with AI?
I'd like to ask you a question. I'm asking in good faith. Here are a few random examples of images people create with AI:
https://civitai.com/images/146193427
https://civitai.com/images/17108902
https://civitai.com/images/63175621
https://civitai.com/images/10703900
https://civitai.com/images/28673784
https://civitai.com/images/59645082
https://civitai.com/images/97017376
https://civitai.com/images/63656891
https://civitai.com/images/99196859
Fundamentally, what makes art, well, art?
Is it the physical act of drawing it manually? Surely not, because generative art[1] (the original non-AI kind) is a widely recognized thing.
Or maybe it's the amount of effort that's spent creating it? Surely not, because paintings as trivial as a simple black or red square are considered art[2].
Or maybe just the very act of self-expression makes something into a piece of art?
Here's Wikipedia's definition:
> Art is a diverse range of human activity centered around works, crafted as an expression of creativity or imagination, which are expected to provide an audience with a worthwhile experience—generally, an experience of emotion, thinking, or beauty.
So is the thesis that, no matter how it's used, and no matter how much effort is spent, an image created with the help of generative AI can never satisfy this definition?
fsloth
No, those are not art in themselves if they are based on dumb prompting.
They are random noise pastiches.
Art requires authorship.
I mean that's my definition :)
The author needs to have sufficient level of artistic skill and talent to be able to produce pieces with artistic merit. Even if they just prompt it - which they likely wont because prompt based outputs alone generally suck.
My definition for sufficient level of skill where AI tools become useful as a form of expression with true artistic merit, would be that you must be able to generate the same image with traditional media without AI:s help
If it turns out there was deep authorship in these works on some level (other than just running through random prompts) then the situation becomes more nuanced.
You should be able to tweak each individual detail with exact authorship. And _understand_ what you are doing.
So if an established painter, photographer or cinematographer uses prompt based tools, there the tools become part of expressive workflow that likely has higher artistic standard.
You can't really develop this sensitivity with prompting alone (generally).
_General_ artistic value is a DIFFERENT thing from doing something a person personally enjoys, though!
It's totally fine to enjoy prompting images.
People enjoy doing puzzles, building legos, practicing piano, playing computer games etc.
Ie engaged in activites which they enjoyu without external valuation mechanism involved.
These things are called _hobbies_. And it's awesome to have hobbies whatever they are.
prettyblocks
I'm more surprised that people think using AI automatically makes image generation not art. AI tooling can be quite complex, and it's much harder than most people think to produce consistent, compelling results, especially with local setups on modest hardware.
nicce
Depends how much involment there is for the end result. Does using traditional, digital paint tools make people artists?
fsloth
You can use AI to transfer content between mediums and manufacture designs.
Succesfull artists don’t necessarily develop the pieces beyond conceptualization and editorial control.
See for instance Damien Hirsch. Or the academic painters with their own workshops.
AI can handle large amounts of routine tasks. Like converting an image to a 3D mesh. I don’t frankly see the difference between that and Damien Hirsch giving his craftsmen a task to deliver a piece.
If you talk to modern art critica you will end up with argumentation where the concept of the piece is what matters not the execution or it’s quality.
If original concept is all that matters very hard to see how AI generated steps could not be part of that pipeline.
Nobody is claiming getting random stuff from generic non-specific prompting counts as art.
But what if you do an original painting or drawing; then have AI convert it to 3D mesh and print it - and pass it off as art? That is a non-obvious situation.
harvey9
Wish my high school art teacher had graded on conceptualisation instead of execution
fsloth
That was because they were training for the "manual labour" part. Which is what most people enjoy and cherish, I think (including myself).
But that's not really how art market values the merchandise.
Conceptualization suffices for the market after sufficient prestige and brand value exists. I guess the Damien Hirsch reference was not that obvious for this venue so -
https://www.businessinsider.com/why-damien-hirst-is-controve...
Note: I'm not saying you should measure the value of art with money. I'm saying those people _who measure the value of art in monetary terms_ may value branding and conceptualization more than manual skill.
trothamel
It's the same sort of people that think they're making art when they take a photograph with their iPhone.
harvey9
My friend who is professional artist argues that no photography is art. If you disregard his view then you need to define where art begins and 'some rando with a camera' ends.
Lammy
> Pixel art is a style, not a format
― person who has never attempted to program for a specific retro system.
Peep the graphics hardware limitations of the Nintendo Famicom/NES, for example: https://noelberry.ca/posts/nes/index.html
You can see these limitations guiding the designs of actual retro games if you know what to look for.
Example: In CAPCOM's Mega Man (1987), the famous “Yellow Devil” boss fight plays the way it does because only the moving chunks of the Devil's body are composed of Sprite tiles. When one of those chunks reaches the opposite side of the screen, that part of the Devil's body becomes drawn with Background tiles instead: https://www.youtube.com/watch?v=8RbPqqPrL_M
This is also why the Devil, Dragon, and Wily boss fights (anything larger than a regular Robot Master) take place entirely in a black void — the Background layer is exhausted drawing the bosses and there's nothing left over for actual play-area background.
__bias
These models always create pixel art that looks like garbage because the people using it don't know enough about the medium to properly judge the output. This article is no exception. If you want pixel art for your project then learn or hire someone with that expertise
sheepscreek
Are you saying people unfamiliar with the 'pixel arts' can't enjoy it their own way?
I don't think the author is posing as a professional. These models are like having a 3D printer at home. Very handy in a pinch, but what they make doesn't replace the real thing.
stymaar
> Are you saying people unfamiliar with the 'pixel arts' can't enjoy it their own way?
The problem isn't that they look shitty to pixel artists or experts, they look shitty for anyone with a bit of observation.
It's junk art the same way Mc Donald's is junk food.
user43928
It's calorie-dense and high in sodium?
I think we could do a bit better here with the explanation of what looks bad to you..
ramblurr
I am just a layperson when it comes to pixel art, but to me the example in the article is clearly garbage/low quality.
- the magenta background is still visible along the edges - the color palette is all over the place - random noise pixels of totally different colors are visible scattered around - the lighting and shading doesn't look that good
stymaar
> It's calorie-dense and high in sodium?
There's plenty of real food that is as heavy and as salted, and as such no healthier, than junk food. But it actually tastes good. (I can't exactly put the finger on what makes an Italian pizza better than a US 7/11 one, but it's not about the former being healthier).
user43928
I'd say it's the fermented, charred dough in combination with fresh, varied toppings.
jrm4
With the exception that it's harder to make the argument that "it's bad for the consumer."
McDonalds', over time, will make you less healthy.
AI slop over time will perhaps change aesthetics, but hard to chalk that up to anything other than personal preference.
orbital-decay
People aren't using it wrong (although it's true most users have no taste), it's just the model devs don't care and generally have no idea about pixel art at all. The model just needs to be trained by someone knowledgeable. For example Retro Diffusion worked pretty well for its era - no mixels, proper palettes, etc.
kouteiheika
> For example Retro Diffusion worked pretty well for its era - no mixels, proper palettes, etc.
That's the wrong layer to tackle this problem. Or I should say: that's necessary, but not sufficient. Yes, the model should be trained to roughly generate something which looks like pixel art (i.e. with a concrete pixel size, reduced/consistent-ish palette, etc.), but enforcing that should be done deterministically in post-processing.
Aerroon
But if the user thinks it looks fine, then won't most players think the same?
I just don't get the complaints about "use proper art" when it comes to games. Most of the time the game has some other, far more glaring, visual issues (anti-aliasing, clipping, icons-not-matching-items).
tuna74
You don't want to anti-alias when having pixel art.
Ohentis
In some circumstances you might not need to anti alias if you are using pixel art. In some circumstances you do. Either way it largely won't hurt.
IsTom
> But if the user thinks it looks fine, then won't most players think the same?
I have a suspicion that when you're looking at (and generating, because you then look at the results) AI generated images and text more you get desensitized and are more likely to say that something looks okay when to other people it won't.
drxzcl
Really? I'm noticing the opposite. First round of imagegen assets seem basically ok, but as time goes on you start to notice the similarities that are properties of the current crop of models and they start to annoy you.
__bias
It's not about what looks fine, it's about having professional standards and understanding the contours of the medium as we know they exist today. Settling is lazy, and these people are hoisting that laziness and lack of concern for visual polish on the public. Instead of getting a high quality product from someone who built up an understanding and expertise, we get poorly considered slop, and more of it too. These people aren't serious about creating a cultural artifact, they just want to get to the end goal as quickly as possible so they can sell it.
Aerroon
Can you explain how every video game has a crazy amount of clipping going on? Is every single developer just lazy?
How come pretty much every modern UI is garbage? Are the devs just lazy? Like google.com on mobile does not have parity with the desktop version (there's no "set specific date range option).
In anything you do, there's a point where you've got to say "okay, this is good enough" and ship it. In fact, I would argue that using pixel art itself is "settling" in the majority of cases.
glimshe
A substantial percentage of AI criticism can boil down to some form of gatekeeping. Maybe AI can't draw a true Scotsman but many will still enjoy the result.
__bias
The problem is that it clogs up the ecosystem of human practitioners that it stole from to exist in the first place.
59percentmore
As soon as artists stop blacklisting devs for using AI placeholders, we can get right on that.
kakapo5672
Isn't that a good thing? Since these models generate garbage output of very low quality, they will never compete with the high quality generated only by human artists. Therefore this is a passing fad and human artists will always dominate.
The worst result would be if these models were any good vs being absolute slop. Then human artists would need to worry. But in fact there is no need to worry at all, as surely people (aside from a few people with bad taste) will clearly see the difference.
perching_aix
how will they tell that the person hired did a good job if they can't tell if the ai cocks up?
__bias
That's the point of hiring an expert isn't it?
perching_aix
would be, yes... are we going to pretend that botch jobs don't exist, and that everyone credentialed actually does an equally (or at all a) good job or something?
you either trust and/or verify. the people clearly cannot verify and clearly do trust. vagueposting about ai bad won't break that vacuous trust, on the contrary, due to the incentives involved, induces distrust on the human expert side.
so maybe you wanna actually elaborate?
ethanj8011
I don't think this told me anything, and it reads like basically something I could have asked claude to whip up for me.
n3ur0n
Had exactly the same reaction. A one shot Claude prompt could have given me this.
syshyk
Which is half of these new websites these days.
jrm4
Great article -- and maybe this is just me -- but I do find the primacy of the online services interesting. Me and my 16gb vram card can get most of this done at home for "free."
duskdozer
When your work depends on people using online LLM services, you probably don't want to encourage people to use their own hardware.
demibabs
Seems like an interesting article but I can’t be bothered to wade through all the Claudish to get to it
JohnQPulp
> Use the same style suffix and seed for every asset in a category so the batch doesn’t drift.
This part's a little confusing to me. What does "suffix" mean in this context? (Is it referring to just like the style-instructions boilerplate text you might include with each image?)
And then also: does locking a set seed really have an effect one way or other for consistency? Wouldn't changing the prompt text even if only slightly have just as much a randomization effect as changing the seed, making fixing it or using a random seed both functionally the same? Or is that not how the latest models work?
Lerc
Are control nets not being used much anymore? I haven't seen anything that provides a similar level of composition control, and yet I do not see them being referenced much at all.
search_facility
They are used a lot, and modern edit models kind of support base cases - depth, canny, openpose - as a reference image. I mean they understand "use image 2 as depth map", result indeed have same depth, etc. But interpretation sometimes arbitrary to model.
For the tasks where several control nets should be adhered (depth and canny and segmentation) dreaded SDXL is still unmatched. SDXL also the only one handling influence weight for each control net normally, this ability completely vanished on modern models
Control nets in early post-SDXL models like ZIT or Flux kind of work too, but still much worse in quality when combined. I suspect pure diffusion architecture of SDXL is really favorable to mixing control signals with precision, DITs probably can be trained to similar level for mixing signals - but it is much harder and never seen in practice. At least every post-SDXL model I tried is trained with single one in mind, judging from my attempts. This rant is from personal expirience.
turbobutton
Related question: Whenever I try to run Image generation locally, be it with Flux or Qwen, I seem to be getting results that are about 1-2 years behind what ChatGPT can produce today - and take up to 5 minutes for a single image. Is this a hardware problem (running it on a decent windows gaming PC), a configuration problem or just the reality of what local models can deliver at the moment on consumer hardware?
FeepingCreature
If you're on a GPU and take five minutes for an image, something is wrong: either you're not actually running on GPU, or you're generating at way too high resolution. Open models work best at not far above 1024x1024.
user43928
I found that for Qwen Image 2.1 the prompt enhancer makes a big difference.
It won't bridge the gap to gpt-image-2.5 though, and it's going to slow down the process further.
With a 4090, Qwen Image 2.1 Turbo takes around 15s per image, depending on the resolution. That's without the prompt enhancer.
Mashimo
With my gamer 4060 ti 16gb it takes around 20 or 30 seconds for flux 1024x1024 image. At least that's how I remember it.
zf00002
Configuration and maybe prompting problem in my experience. Are you doing the prompting or are you letting claude or something do it?
vunderba
> Whenever I try to run Image generation locally, be it with Flux or Qwen, I seem to be getting results that are about 1-2 years behind what ChatGPT can produce today.
At the risk of stating the obvious, you can't really compare local models to proprietary frontier models. For all we know, those models are heavily subsidized and have trillions more parameters than the smaller, modest ones you can run on a single GPU.
I've been running a prompt-adherence benchmark site [1] all the way back since the days of SDXL. If you look at the progress dropdown, I do think the gap is closing.
June of this year saw the release of some of the biggest open-weight bangers (Krea 2, Ideogram 4, and the kind of slept-on Boogu-Image 0.1), though the license terms can be kind of meh.
The best locally hosted model scores 8 out of 15, while the proprietary ones score 12 out of 15 on the text-to-image metric. If you want something that can reasonably decent for local gen while still being relatively small for modest GPUs, take a look at the recently released Moss-Image model [2]. But if you're expecting results at a level of something like GPT-Image-2.5 Sunburst, you're going to be disappointed.
chrisinnes
Thank you for this, I'm a bit behind the times now on local generations and tend to use nano banana pro now when i generate although i used to use stable diffusion and automatic1111 way back, dabbled in everything around and Midjourney 3 onwards. The website shows me clearly where things are at. I expect I'd still like playing with Midjourney for more artistic styled work but it was the cost that drove me away.
vunderba
Midjourney is in kind of an interesting position. They were pretty early in the image generation game. And although they've been vastly eclipsed (even by open-weight local models in terms of prompt adherence), MJ can still be fun to use from an experimental/exploration perspective - especially since you can go all the way back and generate images using the original V1 model from 2022, which is basically a lifetime ago in terms of generative AI
satvikpendem
Look into r/StableDiffusion and follow their prompts and download their ComfyUI workflows.
turbobutton
Thanks, I've done that, but with mixed results. I've also tried comfyui wrappers and alternatives that try to hide the complexity and just set sensible defaults, but I'm beginning to suspect its ultimately a consumer hardware vs million dollar NVidia server question.
satvikpendem
What are you trying to generate? I've also noticed that local models aren't as good as cloud models due to hardware but there are still ways of optimizing the process.
Karuma
You can get results a billion times better than the horrible, disgusting look of ChatGPT in less than a couple of seconds with a model like Krea or MOSS, and ComfyUI. With just a 8GB VRAM GPU or so...
Just visit Civitai, look at the pages for the base models, filter for images with the ComfyUI workflow in them, then you can just download and drag and drop them onto your local ComfyUI app.
ThrowawayTestr
SwarmUI makes things a lot easier. Kroma generates an image in about 90s with a 4070.
tonmoy
I have been asking opus 5.5 to write python to create logos. It is more consistent and easier to control than gen AI for this purpose
doginasuit
I've been using it to generate animated svgs. I made a breakthrough in asking it to create bones/rigging and movement with a python library and apply the result to the svg.
tkdb
How much aesthetic freedom are you giving to opus?
The technical ability of anthropic models continue to steadily improve, but I still find the creative and compositional choices to fall far short of anything I would ever use.
tonmoy
Oh no, I have full creative control. I have to literally specify which fonts to use, where to place a shape etc. But I can ask it to pick a good color palette, put some kind of procedural texture etc. All of this would have taken me a day in photoshop, but with Claude I can give the instructions and let it use Python
praveen9920
One hack I used for generating sprite sheets for a 2d game I’m developing is.. I asked a video generator model to generate a video of required character doing some action and splitting the frames.
Generating grid is cheap but inconsistency creeps up very quickly.
grumpy_coder
All of that still reads as a non-professional messing about at ten different things.
If we want to use a pixel art generator commercially we need 'practical guide' to making a small change to one animation, two months after creating the original character.
We need images generated in psd layers, with actual text using a font.
Sticking to the brand palette religiously.
joe_the_user
My guess would be that "professional" essentially translates to consistent, reliable and structured in the fashion a given industry expects. In this sense, everything AI/LLMs do is going to be "non-professional". Even when they solve frontier level math problems, the mathematicians agree "this is a terrible proof" (this goes against how the field does things). These things is always inconsistent/unreliable/unstructured but sometimes not too inconsistent etc to be useful. "How do use" tip sheets are also going to a list of ad-hoc observations, many of which its easy to get after X many tries with the generator - but that OK, essentially what you want is ways to save yourself time screwing around.
These thing may reach your standard consistent at some point, maybe. It's not important since before and after this point, they will require "messing around" to produce something useful but they can produce something if you need "good enough" but if you demand actually good, they won't arrive ever.
croisillon
Related:
AI-generated posters don’t have to be horrible, 990 comments https://news.ycombinator.com/item?id=49764791
tech_army
Good content. Found it helpful
mochacosine1206
Claude, turn this into a skill.
tkdb
One of the better TLDR's I've seen recently.
westsmith
I wrote this after a few weeks of experimenting with image models through APIs, mostly to work out what’s actually cheap and reliable. Every image in it came from those experiments and cost about a penny each. The bits I found most interesting were why “describe the camera” works better than “describe the view”, and how poorly general models handle exact structure like sprite sheets. Happy to answer questions.
armcat
You mentioned in your README that it was tested with GLM-5.3 but this is not a multi-modal model. Did you mean GLM-5.3-Flash?
Nice article. There's another option for pixel art that's recently been coming into vogue as tool usage improves in LLMs which is procedural graphics.
I actually experimented with creating a set of skills around rendering early AGI style EGA graphical scenes, and it worked surprisingly well.
So rather than storing full bitmaps, you instead store sequences of simple drawing instructions designed to recreate some specific scene - like you would an SVG.
As a result you end up sidestepping a lot of the problems with larger generative models, which can have issues with grid alignment, palette choices, and high-frequency noise.
https://github.com/scpedicini/agi-stylist
https://kq-styles.specr.net