by Jeffrey Braga, Doris Chia-ching Lin

This project is an attempt to use TIRG function for image retrieval with a new furniture style dataset. The original code was published with the following paper:

Composing Text and Image for Image Retrieval - An Empirical Odyssey
Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, James Hays
CVPR 2019. github code source

The dataset was published with the paper below:

Learning Style Compatibility for Furniture
Divyansh Aggarwal, Elchin Valiyev, Fadime Sener, and Angela Yao
CVPR 2018. dataset download

In order to implement TIRG for the furniture style dataset, we have created test_queries.txt to map the index of source image and target image for image retrieval. Each image has captions which can be cross-referenced with testing queries. Here are our dataset stats:

Train Model

• FurnitureStyle: 133346 images
• 56520 unique captions
• Modifiable images 40076

Test Model

• FurnitureStyle: 28812 images
• 28812 test queries

• torchvision
• pytorch
• numpy
• tqdm
• tensorboardX
• Python 3.0.0 or above

• Need to have a cuda compatible graphic card

• main.py: driver script to run training/testing
• datasets.py: Dataset classes for loading images & generate training retrieval queries
• text_model.py: LSTM model to extract text features
• img_text_composition_models.py: various image text compostion models (described in the paper)
• torch_function.py: contains soft triplet loss function and feature normalization function
• test_retrieval.py: functions to perform retrieval test and compute recall performance

Download our generated test_queries.txt from here.

Make sure the dataset include these files:

<dataset_path>/splits/*.txt
<dataset_path>/houzz/<category>/<style>/*.jpeg
<dataset_path>/test_queries.txt`

note that the file name val_split in /splits/ from origin dataset should be renamed to test_val_split

Run training & testing:

python main.py --dataset=furnitureStyle --dataset_path=./furniture-style \
  --num_iters=160000 --model=concat --loss=batch_based_classification \
  --learning_rate_decay_frequency=50000 --comment=furnitureS_concat

python3 main.py --dataset=furnitureStyle --dataset_path=./furniture-style \
  --num_iters=160000 --model=tirg --loss=batch_based_classification \
  --learning_rate_decay_frequency=50000 --comment=furnitureS_tirg