Using AI in your "traditional" artworks

[parkdalegardener]

I haven't "cherry picked the results" as they say. Straight up output from the prompt just to show the workflow and possibilities. If I want to really refine a request (prompt) I run a grid similar to the "contact sheets" and "exposure strips" from the old days of analog print film and darkroom work.

It's not the same thing as the mass generation of images I put out for the Egyptian textures. That is the point and spray technique. Roll the dice and take your chances. A very crude but effective method of generating a bunch of textures all at once.

[parkdalegardener]

So; we've seen we can use AI for making textures and the normals for them.

We can make any environment, though getting perfect environmental lighting is more specific to the software you want to use the environment in.

We can make any background plate we need.

Not really anything you couldn't do in a good paint program. That's the point. AI is simply a tool. Just like Poser or photography or paint.net.

So what else can we do with it? Most of us are familiar with depth maps or z-depth mapping or f-stops in photography. 3D artists have an option to deal with them in their software. Usually they are used to blur out a background or soften near foreground objects. They can even be used to mask out unwanted parts of an image.

2D artists have had no access to such, and after the creation of an image; neither do 3D artists.

Enter MiDaS. MiDaS is an AI model that estimates monocular depth. Huh? It guesses how close or far away something is from a camera. It was developed for autonomous driving. We will use it a little differently. We will use it to estimate the depth in a 2D image.

For instance, I can change out a character at the last second without changing my composition. I generated a portrait of a cougar bard for a game character. At the last second the character wanted to be a fox warrior instead. I'll show you the generation of the bard and then the change to warrior.

The bard image does not have to come from Stable Diffusion. I just show the generation for clarity on the whole thing. In the upper left of the screen you see I used the 2.1 768 model to generate the original image. I then sent my cat picture to image to image tab.

In the second screen grab you can see I have changed to the MiDaS depth mapping model. It has a smaller base size of 512x512 in comparison to the larger 768x768 model used to generate our cat picture.

What I am attempting to illustrate is that the origin of the picture you start with doesn't matter. Just upload it into img2img. Make sure you are using the MiDaS depth mapping. Tell the AI what you really want is a warrior fox. The AI will give you a warrior fox in the same pose (mostly) keeping as much coherence as possible. It works the other way as well to replace the background should you wish.

[parkdalegardener]

Couldn't add the second image for some reason

[sanbie]

Boy that just went right over my head lmao
Will have to read it at a later date in the hope it goes into this old brain lol

[parkdalegardener]

For folk that use a 3D program to create art; there is a depth map render pass that we are all familiar with, though not all actually have a use for it. For people that have 2D art like a photograph or a render of 3D artwork and have no such map for a particular scene; MiDaS can generate one.

Stable Diffusion, dall-e, Midjourney, Wombo, or what have you all have two parts. The first part works by turning words into tokens that represent those words; and the second part then combines the tokens hopefully into something resembling what the user requested. Results are not guaranteed by any stretch of the imagination. That's why it's hard to generate similar images across online services. Each wishes to massage the data behind the scenes. The results from Disco Diffusion look different than Midjourney and so on even though they do the same thing using the same basic model to start with.

Some of the online generation services allow you to choose from the original model files (stable diffusion by Stability AI and partners) and others. I used a MiDaS model to illustrate how powerful depth mapping with this model is. I changed the entire character from cat to dog and outfits from one type to another as easily as shown. Yes image editors and paint programs can do it; but with great effort and skill. You can use auto-masking in Photoshop and other programs. Exactly the same type of thing but you are not training Adobe to get smarter simply by using their ai software.

In fact Adobe now sells an AI plugin that is stable diffusion model. There are also free plugins to use stable diffusion in Blender or the open source Krita. All three can use the MiDaS model. It's kind of auto-magical. I didn't have to generate a mask for anything. Clothing change for instance. I could have left the animal unchanged but the clothing recoloured.

All this is getting more confusing no doubt but it does lead somewhere. I'll have to explain depth mapping with different images. I'm going to get my act together here and be back in a bit.

[parkdalegardener]

Let me try this.

I want to make a picture in my 3D software for the current contest. A couple of statues of the god Horus guarding a path to some pyramids in the distance. One of the options when I render my image is to render a depth map as well. This is true of Blender, Poser, and I am pretty sure; DS.  If I rendered my picture and the depth map and put them side by side they would look like this.

[parkdalegardener]

Pretty straight forward. Your image and a grey scale copy that shows how close to, or far away from; the viewer a thing is. The big statue is close to us in the foreground. Another big statue is in the midground, and some pyramids are in the background.

If we take our depth map and darken up the contrast in paint.net or whatever your paint program/image editor of choice may be; and paint a little black where we want everything to be left alone, we can mask out our two statues from the background.

[sanbie]

Yep, I understand that...

[parkdalegardener]

As you may have noticed by now; stable diffusion has absolutely no idea of Egyptian deities. Horus, Ra, Seth, and so on are all just pharaohs. The mask still needs a bit of work before it's ready for prime time but I can paint a hawk god in place of the statues supplied in my paint program.

If I was starting with a photograph I took on a backpack through Africa trip when I was a teenager, and wanted to use it to supply the background for a separate composition for a contest entry; then the masking to get rid of the foreground would be even more time consuming as you have no depth map to start with.

The MiDaS model was trained for autonomous driving after all so that's the secret. It inputs a 2D colour image from a camera, and as quickly as possible; separates everything it sees from it's surroundings, and then determines how close each object is to becoming a bumper smear and raising your insurance rates.

So MiDaS becomes almost magical for artists. When we make a mask in a paint program we use magic wands and bucket fills, while we enlarger everything so big in order to see each pixel we still leave artifacts along the edge of the mask. MiDaS uses AI to automatically generate more accurate masks than we could using simple descriptions of what we want masked. We don't just didn't see the mask as it's display would just slow everything down.

[parkdalegardener]

Img2img can add detail to an image via depth maps, but it is way more powerful than that. Time to break out paint.net.

I'm going to start another Egyptian themed contest entry. How about a nice caravan of camels crossing the desert? Maybe a few pyramids in the background, and a hot cloudless desert sky. Sounds good; but I don't have a camel figure in my Poser runtime and I'm too slack to model and texture a pyramid in Blender. Why bother? I don't have a camel after all. I do have paint.net. Ok screw it. Paint the top half of a square blue sky. Paint the bottom of the square brown sand. Two yellow triangles for the pyramids. This is going fine. Took less time to paint the background than to type the process. Now for the camels. Three black camels. Beautiful darlings aren't they?