CUDA out of memory? Environments, GPU Issues again

Anyone have any ideas?

It seems my environment was not internally consistent; I tried to sort it out, but now perceptilabs cannot run the textile model and my test code no longer runs unless I include the -previously unnecessary

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
tf.keras.backend.set_session(tf.Session(config=config))

If I don’t run this I get the message

Failed to get convolution algorithm. This is probably because cuDNN failed to initialize

and in the console I see that the dlls were opened OK but then there are other issues with cuBLAS (!)

2021-02-18 14:05:08.505002: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2021-02-18 14:05:08.505264: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2021-02-18 14:05:08.505463: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_100.dll
2021-02-18 14:05:08.505679: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_100.dll
2021-02-18 14:05:08.505856: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_100.dll
2021-02-18 14:05:08.506072: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_100.dll
2021-02-18 14:05:08.506269: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-02-18 14:05:08.506486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-02-18 14:05:08.930693: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-02-18 14:05:08.930798: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0
2021-02-18 14:05:08.931096: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N
2021-02-18 14:05:08.931458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9640 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2021-02-18 14:05:09.952497: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2021-02-18 14:05:10.124863: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2021-02-18 14:05:10.126349: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2021-02-18 14:05:10.126532: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2021-02-18 14:05:10.128726: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-02-18 14:05:10.700777: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2021-02-18 14:05:10.701322: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED

Perceptilabs is getting CUDA_ERROR_OUT_OF_MEMORY errors… more later perhaps