[Bug] IMG2IMG with additional image reference returns wrong image.

[StudenteChamp2]

Git commit

3590aa8

Operating System & Version

Windows 11

GGML backends

Vulkan

Command-line arguments used

sd-cli.exe --backend vulkan0 --diffusion-model flux-2-klein-4b-Q8_0.gguf --llm Qwen3-4B-UD-Q4_K_XL.gguf --vae flux2.full_encoder_small_decoder.safetensors -p "Turn first image into a high quality photograph. Use second image as reference." -r main_image.png -r reference.png -o output_photo.png --cfg-scale 2 --steps 4 --offload-to-cpu --fa --seed 42

Steps to reproduce

Execute command line with the following inputs
main_image.png:

reference.png:

What you expected to happen

A high quality photographs of the main image

What actually happened

Second image(reference.png) is returned!

Logs / error messages / stack trace

D:\sd-MASTER-3590aa8-bin-win-vulkan-x64>sd-cli.exe --backend vulkan0 --diffusion-model flux-2-klein-4b-Q8_0.gguf --llm Qwen3-4B-UD-Q4_K_XL.gguf --vae flux2.full_encoder_small_decoder.safetensors -p "Turn first image into a high quality photograph. Use second image as reference." -r main_image.png -r reference.png -o output_photo.png --cfg-scale 2 --steps 4 --offload-to-cpu --fa --seed 42
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5070 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) Graphics (Intel Corporation) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from D:\sd-MASTER-3590aa8-bin-win-vulkan-x64\ggml-vulkan.dll
load_backend: loaded CPU backend from D:\sd-MASTER-3590aa8-bin-win-vulkan-x64\ggml-cpu-alderlake.dll
[INFO ] common.cpp:2086 - set width x height to 1024 x 1024
[INFO ] stable-diffusion.cpp:407 - loading diffusion model from 'flux-2-klein-4b-Q8_0.gguf'
[INFO ] model_loader.cpp:236 - load flux-2-klein-4b-Q8_0.gguf using gguf format
[INFO ] stable-diffusion.cpp:469 - loading llm from 'Qwen3-4B-UD-Q4_K_XL.gguf'
[INFO ] model_loader.cpp:236 - load Qwen3-4B-UD-Q4_K_XL.gguf using gguf format
[INFO ] stable-diffusion.cpp:483 - loading vae from 'flux2.full_encoder_small_decoder.safetensors'
[INFO ] model_loader.cpp:239 - load flux2.full_encoder_small_decoder.safetensors using safetensors format
[INFO ] stable-diffusion.cpp:534 - Version: Flux.2 klein
[INFO ] stable-diffusion.cpp:560 - Weight type stat: f32: 455 | q8_0: 80 | q4_K: 154 | q5_K: 30 | q6_K: 49 | iq4_xs: 20 | i32: 1 | bf16: 9
[INFO ] stable-diffusion.cpp:561 - Conditioner weight type stat: f32: 145 | q4_K: 154 | q5_K: 30 | q6_K: 49 | iq4_xs: 20
[INFO ] stable-diffusion.cpp:562 - Diffusion model weight type stat: f32: 60 | q8_0: 80 | bf16: 9
[INFO ] stable-diffusion.cpp:563 - VAE weight type stat: f32: 250 | i32: 1
[INFO ] stable-diffusion.cpp:993 - using VAE for encoding / decoding
[INFO ] auto_encoder_kl.hpp:527 - vae decoder: ch = 96
[INFO ] stable-diffusion.cpp:1097 - Using flash attention
[INFO ] stable-diffusion.cpp:1111 - Using flash attention in the diffusion model
[INFO ] stable-diffusion.cpp:1242 - total params memory size = 7822.64MB (VRAM 0.00MB, RAM 7822.64MB): text_encoders 3602.16MB(RAM), diffusion_model 4101.40MB(RAM), vae 119.08MB(RAM), controlnet 0.00MB(N/A), extensions 0.00MB(N/A)
[INFO ] stable-diffusion.cpp:1342 - running in Flux FLOW mode
[INFO ] stable-diffusion.cpp:4573 - generate_image 1024x1024
[INFO ] denoiser.hpp:1079 - get_sigmas with Flux2 scheduler
[INFO ] stable-diffusion.cpp:3579 - sampling using Euler method
[INFO ] stable-diffusion.cpp:4086 - EDIT mode
|##################################################| 108/108 - 656.62MB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 0.20s (read: 0.02s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
[INFO ] stable-diffusion.cpp:4195 - encode_first_stage completed, taking 2.48s
|##################################################| 298/298 - 3.22GB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 0.57s (read: 0.28s, memcpy: 0.00s, convert: 0.02s, copy_to_backend: 0.00s)
[INFO ] stable-diffusion.cpp:4299 - get_learned_condition completed, taking 2.16s
[INFO ] stable-diffusion.cpp:4619 - generating image: 1/1 - seed 42
|##################################################| 149/149 - 6.08GB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 0.66s (read: 0.63s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
|==================================================| 4/4 - 3.39s/it
[INFO ] stable-diffusion.cpp:4651 - sampling completed, taking 16.52s
[INFO ] stable-diffusion.cpp:4665 - generating 1 latent images completed, taking 16.52s
[INFO ] stable-diffusion.cpp:4324 - decoding 1 latents
|##################################################| 140/140 - 527.77MB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 0.20s (read: 0.02s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
[INFO ] stable-diffusion.cpp:4344 - latent 1 decoded, taking 3.15s
[INFO ] stable-diffusion.cpp:4348 - decode_first_stage completed, taking 3.16s
[INFO ] stable-diffusion.cpp:4801 - generate_image completed in 24.32s
[INFO ] main.cpp:489 - save result image 0 to 'output_photo.png' (success)
[INFO ] main.cpp:561 - 1/1 images saved

D:\sd-MASTER-3590aa8-bin-win-vulkan-x64>

Additional context / environment details

CPU:
Intel(R) Core(TM) Ultra 9 285K (3.70 GHz)

GPU:
NVIDIA GeForce RTX 5070 Ti

I get the same result with my CPP implementation:

`
void Renderer::performStableDiffusionRendering()
{
_ASSERT(m_main_image->isFloatingPointImage());
_ASSERT(m_main_image >getNbChannels() == 3);
m_main_image->save(_STRING("D:\SB_Debugging\albedo.png"));

	if (m_secondary_image)
	{
		m_secondary_image->save(_STRING("D:\\SB_Debugging\\albedo_secondary.png"));
		_ASSERT(m_secondary_image_Output);
		performStableDiffusionRenderingInternal(m_secondary_image, m_secondary_image_Output);
		// m_secondary_image_Output->save(_STRING("D:\\SD_Debugging\\_secondary_ouput.png"));
		performStableDiffusionRenderingInternal(m_main_image, m_main_image_Output, secondary_image_Output);
		// m_main_image_Output->save(_STRING("D:\\SD_Debugging\\_primary_ouput.png"));
	}
	else
	{
		performStableDiffusionRenderingInternal(m_main_image, m_main_image_Output);
	}
}

void Renderer::performStableDiffusionRenderingInternal(const std::unique_ptr<Graphics::Texture::RGB32FImage>& input, std::unique_ptr<Graphics::Texture::RGB32FImage>& ouput, const std::unique_ptr<Graphics::Texture::RGB32FImage>& additionalReference)
{
	// Build the stable diffusion Api native input images.
	sd_image_t* nativeInputImages = new sd_image_t[additionalReference ? 2 : 1];

	const auto imageSrc = Graphics::Texture::Helper::create_UNTRACKED_RGB8From(*input);
	{
		nativeInputImages[0].width = getRenderingWidth();
		nativeInputImages[0].height = getRenderingHeight();
		nativeInputImages[0].channel = 3u;
		nativeInputImages[0].data = imageSrc->getMutableRawData();
	}

	const auto imageSrc2 = additionalReference ? Graphics::Texture::Helper::create_UNTRACKED_RGB8From(*additionalReference) : nullptr;
	if (imageSrc2)
	{
		nativeInputImages[1].width = getRenderingWidth();
		nativeInputImages[1].height = getRenderingHeight();
		nativeInputImages[1].channel = 3u;
		nativeInputImages[1].data = imageSrc2->getMutableRawData();
	}

	const std::string modelPathStd = "D:\\AIModels\\";

	// Init the context parameters.
	{
		sd_ctx_params_t sd_ctx_params = {};
		sd_ctx_params_init(&sd_ctx_params);

		const std::string tmp0 = modelPathStd + "flux-2-klein-4b-Q8_0.gguf";
		sd_ctx_params.diffusion_model_path = tmp0.c_str();

		const std::string tmp1 = modelPathStd + "Qwen3-4B-UD-Q4_K_XL.gguf";
		sd_ctx_params.llm_path = tmp1.c_str();

		const std::string tmp2 = modelPathStd + "flux2.full_encoder_small_decoder.safetensors";
		sd_ctx_params.vae_path = tmp2.c_str();

		sd_ctx_params.n_threads = std::max((int)Hardware::HardwareInformationSingleton::instance()->nbOfPhysicalCores(), 4);
		sd_ctx_params.wtype = sd_type_t::SD_TYPE_F16;

		//if (Hardware::HardwareInformationSingleton::instance()->isNvidiaGpu())
		//	sd_ctx_params.backend = "cuda0";
		// else
		sd_ctx_params.backend = "vulkan0";

		sd_ctx_params.flash_attn = true;
		sd_ctx_params.params_backend = "cpu";

		m_stableDiffusionCtx = new_sd_ctx(&sd_ctx_params);
		_ASSERT(m_stableDiffusionCtx != nullptr);
	}

	// Configure generation parameters.

	sd_img_gen_params_t sd_img_gen_params;
	std::string negativePrompt = Utilities::nativeStringToStdString(m_settings.m_negativePrompt);
	replaceAll(negativePrompt, "\n", " ");

	std::string positivePrompt;
	if (additionalReference)
	{
		positivePrompt = "Turn first image into a high quality photograph. Use second image as reference. ";
	}
	else
	{
		positivePrompt = "Turn the image into a high quality photograph. ";
	}

	positivePrompt += Utilities::nativeStringToStdString(m_settings.m_PositivePrompt);
	replaceAll(positivePrompt, "\n", " ");

	{
		sd_img_gen_params_init(&sd_img_gen_params);

		sd_img_gen_params.prompt = positivePrompt.c_str();
		sd_img_gen_params.negative_prompt = negativePrompt.c_str();

		sd_img_gen_params.sample_params.sample_method = EULER_SAMPLE_METHOD;

		sd_img_gen_params.width = getRenderingWidth();
		sd_img_gen_params.height = getRenderingHeight();
		sd_img_gen_params.strength = 0.4f;
		sd_img_gen_params.seed = 42;
		sd_img_gen_params.control_strength = 0.9f;

		sd_img_gen_params.ref_images = nativeInputImages;
		sd_img_gen_params.ref_images_count = additionalReference ? 2 : 1;

		sd_img_gen_params.sample_params.guidance.txt_cfg = 2.0f;
		sd_img_gen_params.sample_params.sample_steps = 4;
	}

	// Set the generation callback.
	sd_set_progress_callback(ProgressCallback, nullptr);

	sd_image_t* result = generate_image(m_stableDiffusionCtx, &sd_img_gen_params);
	if (result)
	{
		_ASSERT(result->width == imageSrc->getWidth());
		_ASSERT(result->height == imageSrc->getHeight());
		_ASSERT(result->channel == imageSrc->getNbChannels());

		// Read back result.
		const uint32_t nbTasks = getRenderingWidth() * getRenderingHeight();
		tbb::parallel_for(size_t(0), size_t(nbTasks), [&](size_t tbbIdx) {

			if (m_stopRender)
			{
				tbb::task::current_context()->cancel_group_execution();
				return;
			}

			const uint32_t pixelLinearIdx = (uint32_t)tbbIdx;
			const uint32_t pixelStartInArray = (uint32_t)tbbIdx * 3;
			const uint32_t pixelPosX = (uint32_t)(pixelLinearIdx % getRenderingWidth());
			const uint32_t pixelPosY = (uint32_t)(pixelLinearIdx / getRenderingWidth());
			const Math::Uvec2 pixelPos = Math::Uvec2(pixelPosX, pixelPosY);

			const uint8_t B = result[0].data[pixelStartInArray];
			const uint8_t G = result[0].data[pixelStartInArray + 1u];
			const uint8_t R = result[0].data[pixelStartInArray + 2u];

			const RGBFColor inputColor = RGBFColor((float)B / (float)IMAGE_UNORM_MAX,
				(float)G / (float)IMAGE_UNORM_MAX, (float)R / (float)IMAGE_UNORM_MAX);
			ouput->setPixelFromPosition(inputColor, pixelPos);
		});


		free(result[0].data);
		free(result);
	}

	free(m_stableDiffusionCtx);
	m_stableDiffusionCtx = nullptr;
	delete[] nativeInputImages;
}

`

[stduhpf]

That's not a bug with the implementation, that's just the model being silly.

[StudenteChamp2]

@stduhpf Thanks or helping! Any idea of how i can fix this and get the same photorealistic results?

[stduhpf]

You could try setting the "main" reference image as init image too, with denoise strength at ~0.8 maybe? Or try with the 9B model, it might do a better job.

[StudenteChamp2]

Tried to put the main image as init image too it does not work.

[StudenteChamp2]

By 9B do you men this model?
https://huggingface.co/black-forest-labs/FLUX.2-klein-base-9B/tree/main

I tried to replace flux2.full_encoder_small_decoder.safetensors by flux-2-klein-base-9b.safetensors. I get those errors:

[ERROR] model_manager.cpp:447 - VAE tensor 'first_stage_model.decoder.conv_in.bias' not in model metadata
[ERROR] model_manager.cpp:447 - VAE tensor 'first_stage_model.decoder.conv_in.weight' not in model metadata
[ERROR] model_manager.cpp:447 - VAE tensor 'first_stage_model.decoder.conv_out.bias' not in model metadata
[ERROR] model_manager.cpp:447 - VAE tensor 'first_stage_model.decoder.conv_out.weight' not in model metadata
[ERROR] model_manager.cpp:447 - VAE tensor 'first_stage_model.decoder.mid.attn_1.k.bias' not in model metadata
[ERROR] model_manager.cpp:447 - VAE tensor 'first_stage_model.decoder.mid.attn_1.k.weight' not in model metadata
[ERROR] model_manager.cpp:447 - VAE tensor 'first_stage_model.decoder.mid.attn_1.norm.bias' not in model metadata
[ERROR] model_manager.cpp:447 - VAE tensor 'first_stage_model.decoder.mid.attn_1.norm.weight' not in model metadata
[ERROR] model_manager.cpp:447 - VAE tensor 'first_stage_model.decoder.mid.attn_1.proj_out.bias' not in model metadata
[ERROR] model_manager.cpp:447 - VAE tensor 'first_stage_model.decoder.mid.attn_1.proj_out.weight' not in model metadata
[ERROR] model_manager.cpp:447 - VAE tensor 'first_stage_model.decoder.mid.attn_1.q.bias' not in model metadata
[ERROR] model_manager.cpp:447 - VAE tensor 'first_stage_model.decoder.mid.attn_1.q.weight' not in model metadata
//...
[ERROR] stable-diffusion.cpp:1175 - model metadata validation failed
[INFO ] main.cpp:791 - new_sd_ctx_t failed

[StudenteChamp2]

@stduhpf Do you know several models that i can try? And that still produces photoreaslistic images. Idealy not too heavy. flux-2-klein-base-9b.safetensors cost is 17GB.
If yes please tell me which variable the model paths should be assigned!

`

	const std::string tmp0 = modelPathStd + "flux-2-klein-4b-Q8_0.gguf";
	sd_ctx_params.diffusion_model_path = tmp0.c_str();

	const std::string tmp1 = modelPathStd + "Qwen3-4B-UD-Q4_K_XL.gguf";
	sd_ctx_params.llm_path = tmp1.c_str();

	const std::string tmp2 = modelPathStd + "flux2.full_encoder_small_decoder.safetensors";
	sd_ctx_params.vae_path = tmp2.c_str();

`
Sorry i a am a noob when it comes to AI!

[wbruna]

"Use second image as reference" is vague. Reference for what? Composition, style, size, color palette, lighting? The more you leave the model free to guess, the more it will guess wrong.

Referring to the images following the syntax on the official examples could also help: one of the examples is “Change image 1 to match the style of image 2”. Or even more specific: "Create a photo realistic depiction of Image 1, keeping its composition. Follow the style and object design of Image 2."

I tried to replace flux2.full_encoder_small_decoder.safetensors by flux-2-klein-base-9b.safetensors. I get those errors:

The same VAE works, but you need to replace the diffusion model and the text encoder; take a look at our official docs. Be sure to check the 9B license, though: it's less permissive than the 4B.

I believe LongCat-Image Edit would be around 6B (found https://huggingface.co/vantagewithai/LongCat-Image-Edit-Turbo-GGUF on a quick search). It would be specified in the same way, but with different LLM and VAE; again, take a look at our official docs.

[StudenteChamp2]

@wbruna Tried the new prompt doesnt work. The official docs has several files in it. Which one should i check?
Please which model(including the file extension) should i try for the following:
`
sd_ctx_params.diffusion_model_path
sd_ctx_params.llm_path
sd_ctx_params.vae_path

`
Thanks for helping!

[StudenteChamp2]

Tried flux-2-klein-base-9b.safetensors as diffusion model in conjunction with Qwen3-8B-UD-Q4_K_XL.gguf as text encoder.
With Vulkan backend i run out of memory on my RTX 5070 Ti .

So i tried on the CPU.. It took 1 hour to render and i get the same result. Main view is ignored and second image(the reference) is returned.

HELP needed! <3

[stduhpf]

Tried to put the main image as init image too it does not work.

Lower the strenth then, until it works. (if you're already under 0.5, it's probably not going to work at all)

[wbruna]

Following up on #1715, since this isn't a bug in the engine.