Img2SceneGraph

Yunhao Ge, Yunkui Peng, Linwei Li, Laurent Itti

University of Southern California

Overview

Img2SceneGraph provides a pipeline that transfers images to scene graphs with node attributes. It can generate graph datasets using on various downstream tasks.

Our primary motive for creating the Img2SceneGraph pipeline, is that it allows rapid idea prototyping for graph compression and representation learning.

A typical work-flow of Img2SceneGraph pipeline contains two steps. For each image:
Step 1 :
Use the pre-trained model from Scene-Graph-Benchmark (Tang, 2020;Tang et al., 2019; 2020) to synthesis the following outputs: 79 bounding boxes (b-boxes)labeled by a single word and over 6,000 relationship pairs (rel-pairs) between b-boxes, both of them are sorted by their corresponding confidence scores.

Step 2 :
Select nodes and edges using different methods to form a Scene Graph:

Select edges first
(a) Select the top n% rel-pairs as edges and corresponding b-boxes as nodes.
(b) Select rel-pairs with confidence score higher than k as edges and corresponding b-boxes as nodes.
(c) Select top m rel-pairs as the edges and corresponding b-boxes as nodes.
Select nodes first
(d) Select the top n% b-boxes as nodes and corresponding rel-pairs as edges.
(e) Select the b-boxes with confidence score higher than k as nodes and corresponding rel-pairs as edges.
(f) Select the top m b-boxes as nodes and corresponding rel-pairs as edges.

For each node, we generate a d-dimension word embedding using (Mikolov et al., 2013)from the label of b-box,which is considered as our initial node feature. If the image has a label, it will be the graph label as well.

Sample Dataset: IMG2SCENEGRAPH-ACSG

Img2SceneGraph-ACSG is a scene graph dataset with graph labels. We created it by using our Img2SceneGraph pipeline on AI Challenger Scene Graph dataset (Jean-Claude).

It is a labeled image dataset that contains 7,120 images with 80 classes. We used method (a) described above withn= 10. The word embedding dimension is 500.
This dataset is used in our paper Graph Autoencoder for Graph Compression and Representation Learning. You can find the download link and the code in the following part.

Download and Source code

You can download theImg2SceneGraph-ACSG dataset here Download link

The code is released! You can freely modify anything you want to get the your own graph dataset. [Code]

If you use our dataset or code, please cite the following paper, thanks!

BibTeX


@inproceedings{ge2021graph,
  title={Graph Autoencoder for Graph Compression and Representation Learning},
  author={Ge, Yunhao and Pang, Yunkui and Li, Linwei and Itti, Laurent},
  booktitle={Neural Compression: From Information Theory to Applications--Workshop@ ICLR 2021},
  year={2021}
}

Related Work

Graph Autoencoder for Graph Compression and Representation Learning
Yunhao Ge*, Yunkui Peng*, Linwei Li and Laurent Itti

[paper] [code] [Img2SceneGraph]