nicolas/abduction-demo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

demo completed

Browse Source
This commit is contained in:
Nicolas Sebastian Schuler
2025-09-10 11:17:53 +02:00
parent a227795a1a
commit c899273827
6 changed files with 212 additions and 180 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@@ -202,3 +202,6 @@ __marimo__/
# Streamlit # Streamlit
.streamlit/secrets.toml .streamlit/secrets.toml
# todos
todos.md

200
.tmp-data/coding_description
View File

@@ -1,158 +1,52 @@
Here's a **Prolog program** that represents the logical rules for classifying an animal as either a **cat** or a **dog**, based on the morphological features described in your detailed comparison. The program uses a set of predicates representing visual traits, and applies a decision framework to infer classification.
This version is designed to work with facts (trait observations) and logical inference using rules.
---
### ✅ Prolog Program: Animal Classifier
```prolog ```prolog
% ------------------------------- % Observable biological traits for identifying cats and dogs
% TRAIT PREDICATES % Based on static image analysis with ranked diagnostic reliability
% These represent observable features from an image.
% Each predicate returns true if the trait is present in the animal.
% -------------------------------
% Skull traits % Muzzle Structure
is_short_rounded_skull :- true. % Assume this is observed (can be set based on image input) short_muzzle.
is_flat_face :- true. rounded_muzzle.
distinct_stop.
straight_nasal_bridge.
narrow_nose.
smooth_rounded_tip.
nose_pad_below_eyes.
pointed_ears.
upright_ears.
narrow_set_ears.
vertical_slit_pupils.
eyes_set_high_on_face.
whiskers_extend_horizontally.
narrow_vertically_aligned_pad.
tail_tapers_to_fine_point.
base_narrower_than_head_width.
head_appears_large.
slender_neck.
compact_body.
is_long_elongated_skull :- false. % Dog traits (secondary confirmation)
is_prominent_muzzle :- false. long_muzzle.
prominent_stop.
convex_nasal_bridge.
larger_nose.
blunt_or_upturned_tip.
nose_pad_prominent.
variable_ear_shapes.
wider_set_ears.
round_pupils.
eyes_level_with_nose_tip.
whiskers_extend_forward_downward.
broad_horizontally_oriented_pad.
tail_blunt_or_bushy.
base_wider_than_head_width.
head_proportionally_smaller.
thicker_neck.
distinct_shoulder_hip_separation.
% Ear traits % Decision rules for classification
is_tall_pointed_ear :- true. dog :- long_muzzle; prominent_stop; convex_nasal_bridge; larger_nose; blunt_or_upturned_tip; nose_pad_prominent; variable_ear_shapes; wider_set_ears; round_pupils; eyes_level_with_nose_tip; whiskers_extend_forward_downward; broad_horizontally_oriented_pad; tail_blunt_or_bushy; base_wider_than_head_width.
is_high_set_ear :- true. cat :- short_muzzle; rounded_muzzle; distinct_stop; straight_nasal_bridge; narrow_nose; smooth_rounded_tip; nose_pad_below_eyes; pointed_ears; upright_ears; narrow_set_ears; vertical_slit_pupils; eyes_set_high_on_face; whiskers_extend_horizontally; narrow_vertically_aligned_pad; tail_tapers_to_fine_point; base_narrower_than_head_width.
is_floppy_or_rounded_ear :- false. % Queries for classification
is_wide_spaced_ear :- false. query(dog).
query(cat).
% Eye traits ```
is_large_round_eye :- true.
is_lateral_eye_position :- true.
is_small_eye :- false.
is_forward_facing_eye :- false.
% Paw traits
is_small_paw_with_visible_toes :- true.
has_retractable_claws :- true.
is_large_paw_with_non_retractable_claws :- false.
% Tail traits
is_long_thin_tail :- true.
is_tail_held_high_or_curled :- true.
is_thick_tail :- false.
% Body posture traits
is_low_to_ground_body :- true.
is_compact_body :- true.
is_robust_upright_body :- false.
is_long_legs_relative_to_body :- false.
% Muzzle traits (profile)
has_visible_stop_in_profile :- false.
has_overbite_or_underbite :- false.
% -------------------------------
% CLASSIFICATION RULES
% These are based on the decision framework provided.
% -------------------------------
% Cat classification rules
is_cat :-
cat_traits_count(CatCount),
dog_traits_count(DogCount),
CatCount >= 3,
DogCount =:= 0.
% Dog classification rules
is_dog :-
cat_traits_count(CatCount),
dog_traits_count(DogCount),
DogCount >= 3,
CatCount =:= 0.
% Counting cat-specific traits
cat_traits_count(Count) :-
findall(Trait, cat_trait(Trait), Traits),
length(Traits, Count).
% Define which predicates are cat-specific traits
cat_trait(is_short_rounded_skull).
cat_trait(is_flat_face).
cat_trait(is_tall_pointed_ear).
cat_trait(is_high_set_ear).
cat_trait(is_large_round_eye).
cat_trait(is_lateral_eye_position).
cat_trait(is_small_paw_with_visible_toes).
cat_trait(has_retractable_claws).
cat_trait(is_long_thin_tail).
cat_trait(is_tail_held_high_or_curled).
cat_trait(is_low_to_ground_body).
cat_trait(is_compact_body).
% Counting dog-specific traits
dog_traits_count(Count) :-
findall(Trait, dog_trait(Trait), Traits),
length(Traits, Count).
% Define which predicates are dog-specific traits
dog_trait(is_long_elongated_skull).
dog_trait(is_prominent_muzzle).
dog_trait(is_floppy_or_rounded_ear).
dog_trait(is_wide_spaced_ear).
dog_trait(is_small_eye).
dog_trait(is_forward_facing_eye).
dog_trait(is_large_paw_with_non_retractable_claws).
dog_trait(is_thick_tail).
dog_trait(is_robust_upright_body).
dog_trait(is_long_legs_relative_to_body).
dog_trait(has_visible_stop_in_profile).
dog_trait(has_overbite_or_underbite).
% -------------------------------
% SAMPLE QUERY
% To test, you can run:
% ?- is_cat.
% ?- is_dog.
% You may want to redefine facts like is_short_rounded_skull/0, etc., depending on image input.
% -------------------------------
```
---
### 🧠 How It Works
- **Traits** are defined as simple predicates (e.g., `is_short_rounded_skull`, `is_large_round_eye`) that can be set to `true` or `false`.
- The program counts how many traits match for cats and dogs.
- If **three or more cat-specific traits** are present, and **no dog-specific traits**, it classifies as a **cat**.
- Similarly, if three or more **dog-specific traits** are present, it classifies as a **dog**.
---
### 🧪 Example Queries
```prolog
?- is_cat. % Returns true if enough cat traits are set to true
?- is_dog. % Returns true if enough dog traits are set to true
```
---
### 🛠️ Extending for Image Input
To use this with real image data (e.g., a computer vision pipeline), you would:
- Set the facts dynamically from image analysis.
- Use `assert/1` or `retract/1` to change trait values.
- For example:
```prolog
?- assert(is_short_rounded_skull).
?- is_cat.
```
Let me know if you want a version that integrates with actual image recognition libraries or uses probability-based reasoning!

BIN
.tmp-data/dog.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 408 KiB

14
.tmp-data/grounding_results Normal file
View File

@@ -0,0 +1,14 @@
- 0.9:long_muzzle
- 0.9:prominent_stop
- 0.9:convex_nasal_bridge
- 0.9:larger_nose
- 0.9:blunt_or_upturned_tip
- 0.9:nose_pad_prominent
- 0.9:variable_ear_shapes
- 0.9:wider_set_ears
- 0.9:round_pupils
- 0.9:eyes_level_with_nose_tip
- 0.9:whiskers_extend_forward_downward
- 0.9:broad_horizontally_oriented_pad
- 0.9:tail_blunt_or_bushy
- 0.9:base_wider_than_head_width

35
.tmp-data/reasoning_description Normal file
View File

@@ -0,0 +1,35 @@
### Comparative Analysis: Cat vs. Dog (Image-Based Identification)
*Based solely on observable biological traits from a static image. Traits are ranked by diagnostic reliability.*
1. **Muzzle Structure**
- *Cat*: Short, rounded muzzle with a distinct "stop" (forehead indentation), nasal bridge straight/rounded.
- *Dog*: Longer muzzle (proportional to head size), prominent stop (dip between eyes/nose) common in breeds; nasal bridge often convex.
2. **Nose Shape & Position**
- *Cat*: Small, narrow nose with a smooth, rounded tip; nose pad positioned directly below eyes.
- *Dog*: Larger nose relative to head; tip often blunt or slightly upturned; nose pad may appear more prominent.
3. **Ear Shape & Placement**
- *Cat*: Pointed, upright ears; placed high on head with minimal distance between ears (narrow-set).
- *Dog*: Variable ear shapes (floppy, erect, semi-erect); typically wider-set (greater inter-ear distance).
4. **Eye Morphology**
- *Cat*: Vertically slit pupils (even when dilated); eyes set closer to muzzle, higher on face.
- *Dog*: Round pupils (except in rare breeds like Huskies); eyes usually level with nose tip.
5. **Whisker Pad Position**
- *Cat*: Whiskers extend horizontally from a narrow, vertically aligned facial pad.
- *Dog*: Whiskers often extend forward and downward; facial pad broader, more horizontally oriented.
6. **Tail Structure**
- *Cat*: Tail typically tapers to a fine point; base narrower than head width.
- *Dog*: Tail often blunt or bushy at tip; base usually wider than head width (except in breeds like Greyhounds).
7. **Body Proportions (Head-to-Body)**
- *Cat*: Head appears relatively large; neck slender, body compact with less visible separation between shoulders/hips.
- *Dog*: Head proportionally smaller; neck often thicker; distinct shoulder/hip separation visible in stance.
*Decision Logic*:
- **Muzzle + nose** (Trait 1 & 2) are the most reliable initial identifiers. A short muzzle with rounded nasal bridge confirms cat; elongated muzzle with prominent stop suggests dog.
- **Eyes + ears** (Trait 3 & 4) provide secondary confirmation (e.g., vertical pupils override ambiguous muzzle).
- *Note*: Breed variations exist (e.g., Pugs have shortened muzzles), but core traits remain consistent at species level.

140
main_xai.py
View File

@@ -2,26 +2,32 @@ from pathlib import Path
from langchain_core.output_parsers import StrOutputParser from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate from langchain_core.prompts import ChatPromptTemplate
from langchain_ollama.llms import OllamaLLM from langchain_openai import ChatOpenAI
from problog import get_evaluatable
from problog.program import PrologString
from src.data import LabeledImage, load_data from src.data import LabeledImage, load_data
from src.img_utils import encode_base64_resized from src.img_utils import encode_base64_resized
TESTING = 1 TESTING = 1
EPSILON_PROB = 0.01
def reasoning(): def reasoning():
template_reasoning = ChatPromptTemplate.from_messages([ template_reasoning = ChatPromptTemplate.from_messages([
("system", "{role_reasoning}"), ("system", "{role_reasoning}"),
("human", "Question: {question_reasoning}"), ("human", "Question: {question_reasoning}"),
]) ])
model_reasoning = ChatOpenAI(
model_reasoning = OllamaLLM(model="hf.co/unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_M") model="qwen3-thinking",
base_url="http://localhost:8080/v1",
)
reasoning_chain = template_reasoning | model_reasoning | StrOutputParser() reasoning_chain = template_reasoning | model_reasoning | StrOutputParser()
description = reasoning_chain.invoke({ description = reasoning_chain.invoke({
"role_reasoning": "You are an expert in the classification of whether an animal is a cat or a dog", "role_reasoning": "You are a scientific expert in the classification of whether an animal is a cat or a dog. If tasked to answer questions, you shall adhere to scientific facts, think step-by-step, and explain your decision-making process. Focus on 'why' something is done, especially for complex logic, rather than *what* is done. Your answer should be concise and direct, and avoid conversational fillers. Format your answer appropriately for better understanding.",
"question_reasoning": "I want you to do a comparative analysis of cats and dogs. Your analysis must use the inherent traits and biological characteristics of each species. You should list each of these characteristics so that an informed decision can be made about whether a given animal, e.g., in the form of an image, is a cat or a dog. Please provide a detailed analysis, focusing on traits and characteristics that can be extracted from a given image.", "question_reasoning": "I want you to do a comparative analysis of cats and dogs. Your analysis must use the inherent traits and biological characteristics of each species. You should list each of these characteristics so that an informed decision can be made about whether a given animal depicted in an image is a cat or a dog. Please provide a detailed analysis, focusing on traits and characteristics that can be extracted from a given image. For formatting please use a list-like fashion.",
}) })
return description return description
@@ -31,25 +37,33 @@ def coding(description: str):
("system", "{role_coding}"), ("system", "{role_coding}"),
("human", "Instructions: {instruction}\n Description: {description}"), ("human", "Instructions: {instruction}\n Description: {description}"),
]) ])
model_coding = OllamaLLM(model="hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_XL") model_coding = ChatOpenAI(
model="qwen3-coder",
base_url="http://localhost:8080/v1",
)
coding_chain = template_coding | model_coding | StrOutputParser() coding_chain = template_coding | model_coding | StrOutputParser()
coding_description = coding_chain.invoke({ coding_description = coding_chain.invoke({
"role_coding": "You are an expert Prolog programmer with extended knowledge in reasoning and probabilities. Given instructions and a description, you can write a correct Prolog program that expresses the given task as a suitable logical program in Prolog", "role_coding": """You are an expert Prolog programmer with extended knowledge in reasoning and probabilities. Given instructions and a description, you can write a correct Prolog program that expresses the given question as a suitable logical program in Prolog. You shall format your answer so that it can be directly used as an input for a Prolog interpreter. Do not incorporate example facts or queries into the knowledge base; these will be added later by the user. If necessary, add comments to your program to provide explanations to the user. The proposed facts should follow the form:
"instruction": "Write a logical program for the following description", - <trait>.
There shall only be two rules of the following form present at the end:
- dog :- <trait1>; <trait2>; ...; <traitN>.
- cat :- <traitA>; <traitB>; ...; <traitX>.
Make sure to use ';' and not ',' for these two rules!
Lastly, the following two queries should be added:
- query(dog).
- query(cat).
""",
"instruction": "Write a logical program for the following description:",
"description": description, "description": description,
}) })
return coding_description return coding_description
def grounding(coding_description: str, labeled_image: LabeledImage): def grounding(coding_description: str, labeled_image: LabeledImage):
model_vl = OllamaLLM(model="qwen2.5vl:7b") model_vl = ChatOpenAI(
model_vl_ctx = model_vl.bind( model="minicpm-v-45",
images=[ base_url="http://localhost:8080/v1",
encode_base64_resized(
Path(".tmp-data/highlight_spotlight_heatmap.jpg"), max_width=512, max_height=512, quality=70
)
]
) )
template_grounding = ChatPromptTemplate.from_messages([ template_grounding = ChatPromptTemplate.from_messages([
@@ -61,40 +75,112 @@ def grounding(coding_description: str, labeled_image: LabeledImage):
Description: {description} Description: {description}
""", """,
), ),
("placeholder", "{image}"),
]) ])
grounding_chain = template_grounding | model_vl_ctx | StrOutputParser() grounding_chain = template_grounding | model_vl | StrOutputParser()
return grounding_chain.invoke( return grounding_chain.invoke(
{ {
"role_vl": "You are an expert in analyzing an image to extract and match features of a given list.", "role_vl": "You are an expert in analyzing images to extract and match features of a given list. First, you look at the list of given features (facts written in Prolog), and then you analyze the given image for these features. If you are uncertain whether a feature matches, please acknowledge this and inform the user, but do not add the feature to the list of matched features. Please follow the user's instructions precisely.",
"instruction": """You are given a logic program in the following description and an image with a heatmap as input. Your task is to do the following steps: "instruction": """You are given a logic program in the following description and an image. Your task is to do the following steps:
1. Extract the list of features from the given Prolog program that contribute to deciding whether the image is a cat or a dog. 1. Extract the list of features/facts from the given Prolog program that contribute to deciding whether the image is a cat or a dog.
2. Match only the features highlighted by the heatmap in the given image with the features you retrieved from the Prolog program. Give a likelihood of how sure you are with your matching. Print your result in this format: 2. Match only the features that are highlighted in the given image with the features (Prolog facts) you retrieved from the Prolog program. If no highlighting is visible consider the whole image. Give a likelihood, as decimal number, of how sure you are of your match. Print your result in this format:
- <feature from prolog program>: <likelihood> - <likelihood>:<trait1>
- <likelihood>:<trait2>
- ... - ...
""", """,
"description": coding_description, "description": coding_description,
"image": [
(
"human",
[
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpg;base64,{encode_base64_resized(Path('.tmp-data/dog.jpg'), max_width=512, max_height=512, quality=70)}"
},
}
],
)
],
}, },
) )
def execute_logic_program(coding_description: str, grounding_results: str):
program = coding_description
# remove code-block notation
program = program.split("\n", 1)[-1]
program = program.rsplit("\n", 2)[0]
# extract evidence from grounding
evidence = []
for grounds in grounding_results.splitlines():
# e.g., '- 0.95:visible_nose_ride'
prob, fact = grounds[2:].split(":", maxsplit=2)
evidence.append((fact, prob))
# add probabilities
program_lines = program.splitlines()
for idx, line in enumerate(program_lines):
if len(line) <= 1 or line.startswith("%"):
continue
else:
if line.startswith("cat :-") or line.startswith("dog :-") or "query" in line:
continue
else:
matched = False
for fact, prob in evidence:
if fact in line:
program_lines[idx] = f"{prob}::" + line
matched = True
if not matched:
program_lines[idx] = f"{EPSILON_PROB}::" + line
program_sanitized = "\n".join(program_lines)
# evaluate logical program
print(program_sanitized)
result = get_evaluatable().create_from(PrologString(program_sanitized)).evaluate()
# get final probabilities
p_cat, p_dog = (0.0, 0.0)
for term in result.keys():
if str(term) == "dog":
p_dog = result[term]
elif str(term) == "cat":
p_cat = result[term]
else:
raise KeyError("Unknown key encountered!")
return p_cat, p_dog
def main(): def main():
print("Starting Abduction Demo") print("Starting Abduction Demo")
labeled_images = load_data() labeled_images = load_data()
labeled_image = labeled_images[1] labeled_image = labeled_images[1]
# image = labeled_image.image
# mask = labeled_image.create_mask([labeled_image.labels[0]])
if TESTING == 1: if TESTING == 1:
reasoning_description = Path(".tmp-data/reasoning_description").open("r").read()
coding_description = Path(".tmp-data/coding_description").open("r").read() coding_description = Path(".tmp-data/coding_description").open("r").read()
grounding_results = Path(".tmp-data/grounding_results").open("r").read()
else: else:
coding_description = coding(reasoning()) reasoning_description = reasoning()
with open(".tmp-data/reasoning_description", "w") as f:
f.write(reasoning_description)
coding_description = coding(reasoning_description)
with open(".tmp-data/coding_description", "w") as f: with open(".tmp-data/coding_description", "w") as f:
f.write(coding_description) f.write(coding_description)
result = grounding(coding_description, labeled_image) grounding_results = grounding(coding_description, labeled_image)
# TODO: Feed this into the Prolog program and execute to reach final verdict with open(".tmp-data/grounding_results", "w") as f:
print(result) f.write(grounding_results)
print(grounding_results)
p_cat, p_dog = execute_logic_program(coding_description, grounding_results)
print(f"Cat Probability: {p_cat}")
print(f"Dog Probability: {p_dog}")
print("End Abduction Demo") print("End Abduction Demo")