Limit RAM Usage of GPU/s

[apdullahyayik]

By default, tensorflow 1.14 alloctates whole RAM in GPU/s even though they are not needed in run-time. For ranker task, in this library, if indexing lib that uses GPU (for instance through faiss-gpu) is to be deployed, memory problem like below is faced.

RuntimeError: Error in void faiss::gpu::allocMemorySpaceV(faiss::gpu::MemorySpace, void**, size_t) at gpu/utils/MemorySpace.cpp:26: Error: 'err == cudaSuccess' failed: failed to cudaMalloc 1610612736 bytes (error 2 out of memory)

I have added config.gpu_options.per_process_gpu_memory_fraction = 0.7 configuration for session creation at algorithm/base.py. By doing this, RAM usage of GPU/s is explicitely limited during training and 0.3 of them are reserved for indexing. Of course, there are draw-backs, is there another path to go you think?

def _sess_config(self, tf_sess_config=None):
    if not tf_sess_config:
        # Session config based on:
        # https://software.intel.com/content/www/us/en/develop/articles/tips-to-improve-performance-for-popular-deep-learning-frameworks-on-multi-core-cpus.html
        tf_sess_config = {
            "intra_op_parallelism_threads": 0,
            "inter_op_parallelism_threads": 0,
            "allow_soft_placement": True,
            "device_count": {"CPU": self.cpu_num}
        }
    #    os.environ["OMP_NUM_THREADS"] = f"{self.cpu_num}"

    config = tf.ConfigProto(**tf_sess_config)

    # limit GPU Usage
    config.gpu_options.per_process_gpu_memory_fraction = 0.7
    return tf.Session(config=config)

[massquantity]

I think the best way is making tensorflow allocate GPU memory dynamically, but I don't know how to achieve this...