About

Challenge

Currently, there is a rapid growth in the number and volume of image storages, especially in the Internet. Throughout the images' lifecycle, it’s transformed by users and services. Changing of image properties, taking into account Big Data problems, duplication and presence of similar images, leads to inefficient work of image based search services (IBSS) and image storage management systems (ISMS).

Proposal

The Next Generation of Image Based Search Technology (IBST) is developing to solve these problems. Proposed IBST is based on the use of a new unified hybrid image model (UHIM). The most efficient computational intelligence, data mining and computer vision methods are used to identify UHIM. Its main advantages: new object-oriented image description, new invariant features to compensate image transformations, new fixed-length template (FLT) for storage an image features


FLT is needed for quick image comparison and search. Due to these and some other solutions, technologies you can effectively (according to speed, quality, etc.) search for images in the Internet and Big Image Storages in conditions of image duplication, presence of a large number of similar images and various image transformations.

Outcomes

In general, the project is part of a global initiative to create a new generation Internet and support an open and safety information society. Our goals are search of new ideas, research and development of new innovative solutions and tools for improving search in the Internet and Big Data Storages. Now we are focused on development of Image Based Search Technology for the Internet and Image Storages. Of course, our main goal is to help people and benefit society!

Research

Unified Hybrid Image Model

The basic element of the project's novelty is the unified hybrid image model (UHIM), which consists of the following complementary components.

First UHIM Component

A unified set of photometric image features that are estimated using machine vision algorithms. Most features are invariants for common image transformations (file format change, tonal correction, quantization, sharpening, scaling, etc.). Compared to analogues, the use of such features makes it possible to efficiently search for transformed images and adequately distinguish them from similar images. Author's technology. Such features are indispensable when there are no objects in the image that can be identified using computational intelligence algorithms.

Second UHIM Component

A unified set of image features that are estimated using the CNNs. On the basis of these features, according to the author's technique, an object-oriented description of the image is compiled. Which is very effective for searching in the presence of standard objects in the image. The list of found objects can be added to the metadata for quick search by words.

Third UHIM Component

In the future, it is planned to add into the model a new technology for local feature (LF) detecting and storing. Which additionally classifies and filters LFs before storing.

Compared to analogues, this approach will leave only the required minimum of significant LFs and provide real-time search. Such LFs are needed to detect objects of a unique shape, objects / images that are inserted in other images. Which is relevant, for example, to search images under copyright protection program.

Outcomes

Compared to analogs, the components of the model complement each other to efficiently search for images in various conditions of use, taking into account the context, transformations, the presence of similar images and Big Date problems.

Technology

For quick search, the image features are estimated once when image placed in the storage. Image features are stored as a relatively small FLT. The search is performed on the basis of comparison and estimation of the similarity measure of the FLTs; because the FLTs are relatively small, the search is very fast. The search result is presented as a list of images from the storage, sorted in descending order by the measure of similarity with the searched image.
To implement this technology, innovative FLT model is proposed. On the basis of this FLT model, in the future, a model of a search hypercube will be built for effective search in Big Date conditions.

Outcomes

Using the proposed models and technology allows work effectively with any image storages (lake, warehouse, photobank, archive, etc.). Since the data in the image storage and its structure do not change. A small warehouse is created for each image storage, where image accounts are stored, including image FLTs and links.
Search based on the proposed model and technology assumes that instead of images, we will operate (compare, transmit over the network, etc.) with FLTs and image links. The size of which is several orders of magnitude smaller than the size of the images. In such a situation, it’s possible to significantly reduce the hardware requirements for servers and network equipment by providing real-time search. Even for Big Data.

Experiments

Description

Currently, we are building and testing the implementation effectiveness of the first UHIM component. To achieve this goal we do next steps.
We use a dataset of more than 100 000 images of various formats and classes. We take an image from this dataset and transform it in various ways. We change the scale, sharpen, perform gradational correction, rotation, etc. The transformed images are added to the dataset / storage. For all images in the storage, FLTs are built (only the first component). We start the process of searching for the original image. As a result of the search, the top 12 of the sorted list of images are displayed according to the decreasing measure of similarity (variance) with the original image. The original image is always the first (because variance = 0). For experiments, we have developed a Python Prototype of image based search service. Search examples are shown below (top right - original image; left - search results).

Results & Discussion

The purpose of such experiments is to test the model invariance to transformations and to estimate the time. As a result of tests, it was found that for common transformations, modified images are correctly displayed in the top 12 of the search results immediately after the original image. An unoptimized IBSS Prototype on a standard user laptop processes with dataset of 100 000 images works in 1.7 seconds. According to our estimates the optimized multithreaded C ++ program will be an order of magnitude faster. This is possible due to the use of small FLTs. For significant progress in time in big data conditions (in addition to efficient local brute force of FLTs), a search technology using a hypercube of image parameters will be proposed.




Ecosystem

Description

For successful promotion of the proposed model and technology, it is planned to develop new Image Based Search Service (IBSS). Which can be embedded in browsers, search services, image storage management systems. The IBSS prototype will be developed in the project.
IBSS (like a browser with a search line) is oriented on ordinary Internet user, for every modern person who works with photo archives. The IBST will be published in the public/free access. The results will be discussed at international seminars and conferences. Thanks to innovative features, mostly, the service will be promoted by Internet users and software developers which are focused on working with images. Starting with browsers.

Perspectives

The operation of the service is provided by libraries with AI, Data Mining and Computer Vision algorithms. Thereby, IBSS has a huge development potential (through the implementation of additional capabilities based on a microservice architecture). Here are just a few examples.
Image similarity analysis can be used to create collections, clean up storage from duplicates, and efficiently perform other storage management operations (especially for Big Data).
Using CNNs allows to extract information about objects from an image (find people's, faces, recognize text, etc.), including their images, which is useful for various purposes; add an information (keywords, etc.) about objects to the file metadata. To integrate image based search and word based search in future image search service "All in one".
Using local feature detectors allows searching for unique images, objects / images in other images.

Contacts



Project supervisor
ScD, Professor Kyrylo Smelyakov
kyrylo.smelyakov@nure.ua
https://nure.ua/en/staff/kyrylo-smelyakov
Software Engineering Department,
Kharkiv National University of Radio Electronics (NURE)
General legal address: Nauky Ave. 14, Kharkiv, 61166, Ukraine