Classification_Notes of XML4LULC Script

Some notes while studying script - XML4LULC

XG Boosted

Error

ValueError: Invalid classes inferred from unique values of y. Expected: [0 1 2 3 4 5], got [1 2 3 4 5 6]

这是因为类列必须从 0 开始（从 1.3.2 版本开始就需要）。解决这个问题的简单方法是使用 sklearn.preprocssing 库中的 LabelEncoder1。

Encode target labels with value between 0 and n_classes-12.

1
2
3
4
5
6
7
8
9
10
from sklearn import preprocessing

le = preprocessing.LabelEncoder()
le = le.fit(tr_label)
le_tr_label = le.transform(tr_label)
print(le_tr_label)

le = le.fit(val_label)
le_val_label = le.transform(val_label)
print(le_val_label)

Import “tensorflow.keras” could not be resolved

细心观察，发现，和最初代码相比，交换了顺序后，第二句是通过的（之前忽略了），其实只是第三句报错。那赶快验证一下tensorflow中keras的版本和位置，可以看到tensorflow里的keras版本号有个tf标志，再看路径，多了一个python的中间路径！于是进行后续验证，加入中间的python路径后，成功运行3。

1
from tensorflow.python.keras import layers

ImportError: cannot import name ‘BatchNormalization’ from ‘tensorflow.python.keras.layers’

使用的是之前老的tf.keras导入。Layers现在可以直接从tensorflow.keras.layers导入4。

Replace this line from tensorflow.python.keras.layers import BatchNormalization with this one from keras.layers.normalization.batch_normalization import BatchNormalization5

Change imports

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
from tensorflow.python.keras.layers import Input, Dense, Flatten, Dropout,Concatenate, LSTM, GRU, SimpleRNN,BatchNormalization,\
 Bidirectional, Reshape, MaxPool2D,AveragePooling3D, MaxPooling3D,MaxPooling2D,TimeDistributed, Conv2D, Conv3D

# Change to this
import tensorflow as tf
import keras
import time
from tensorflow import keras
from tensorflow.python.keras import regularizers
# from keras.layers.convolutional import Conv2D, Conv3D 
from tensorflow.python.keras.models import Model,Sequential
from keras.layers import Input, Dense, Flatten, Dropout,Concatenate, LSTM, GRU, SimpleRNN, Reshape, MaxPool2D,AveragePooling3D, MaxPooling3D,MaxPooling2D, Conv2D, Conv3D, Bidirectional, TimeDistributed
from keras.layers.normalization.batch_normalization import BatchNormalization
from keras.layers import CuDNNLSTM,CuDNNGRU
from tensorflow.python.keras.backend import squeeze
from keras.constraints import maxnorm
from keras.optimizers import SGD, Adam, RMSprop, Adagrad, Adadelta, Adamax, Nadam
from tensorflow.python.keras.regularizers import l2
#from tensorflow.python.keras.layers import Dense, Dropout, Flatten, LSTM, GRU, SimpleRNN, Bidirectional, Reshape
# from tensorflow.python.keras.layers import Dense

from tensorflow.python.keras.layers import Activation, ReLU, Add
# from keras.layers.convolutional import MaxPooling2D
# from keras.layers.normalization import BatchNormalization
from keras.utils import np_utils
from keras.utils import plot_model

Conda Error: conda solving environment failed

  1. Change source channel:

    • Open C:\Users\mengy\.condarc in notepad

    • edit

    1
    2
    3
    4
    channels:
        - conda-forge
        - defaults
    channel_priority: strict

    to

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    channels:
        - https://mirrors.ustc.edu.cn/anaconda/pkgs/main/
        - https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge/
        - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
        - https://mirrors.bfsu.edu.cn/anaconda/pkgs/main/
        - https://mirrors.bfsu.edu.cn/anaconda/pkgs/free/
    show_channel_urls: true
    ssl_verify: true
    channel_priority: flexible

  2. conda update -n base conda

  3. Use new environment

    1
    conda activate dl_lulc

No module named ogr and gdal

在GDAL库的网站（https://www.lfd.uci.edu/~gohlke/pythonlibs/#gdal）找到符合自己配置的安装包，cp表示Python版本，我的是Python3.7，故选择cp37；64位，32位，需要与自己电脑位数对应11.

1
pip install C:\Users\mengy\Downloads\Documents\GDAL-3.3.3-cp310-cp310-win_amd64.whl

The Python modules that gisinternals.com deliver are not up to date. The error messages in your question are showing where the error happens. For example this one that refers to line 1920 in file gdal.py12.

1
from osgeo import ogr, gdal

model.summary() not showing layers information

And ImportError: cannot import name 'models' from 'tensorflow.python.keras.models'

The issue happens when mixing tensorflow with standalone keras. This is because the keras package is now part of the tf and no more a standalone package13.

I just resolved this issue by ensuring that all usages of keras were replaced with tensorflow.keras in my imports. It appears that breaking backwards compatibility has happened hard and fast. In my case, this was caused by from keras.models import Model. I replaced it with from tensorflow.keras.models import Model, and the problem stopped14.

Before python codes:

1
2
3
4
5
6
7
8
9
from tensorflow.keras import layers, Model, Input

input = Input(cube_shape, name='input')

model = Model(inputs=input, outputs=Output)

input = Input(input_shape, name='input')

model = Model(inputs=input, outputs=Output)

Changed python codes:

1
2
3
4
5
6
7
8
9
10
11
12
# VGG16
import tensorflow as tf

input = tf.keras.Input(cube_shape, name='input')

model = tf.keras.models.Model(inputs=input, outputs=Output)

# ResNet50

input = Input(input_shape, name='input')

model = tf.keras.models.Model(inputs=input, outputs=Output)

Model Summary

InternalError: Failed copying input tensor

Solutions: TensorFlow: Dst tensor is not initialized - Stack Overflow15.

主要原因在于 batch_size 太大，内存无法负载，将 batch_size 适当调小即可正常运行。 默认情况下，TF 会尽可能地多分配占用 GPU 内存，通过调整 GPUConfig 可以设置为按需分配内存15.

另外，使用 Jupyter Notebook 进行长期模型训练时，可能由于 GPU 内存无法及时释放导致该报错16。

Tensorflow install GPU & CUDA

  1. Check PC support for GPU

    • ctrl + alt + delete - Task Manager - Performance

    Support for GPU

  2. GPU version

    • NVIDIA Control Panel - Home - Version: 511.65

    • NVIDIA GeForce RTX 3050Ti Laptop GPU

  3. CUDA version for GPU

    • The version of the development NVIDIA GPU Driver packaged in each CUDA Toolkit release is shown below6.

    • In Yellow bracket we choose Red CUDA 11.2.0 GA

    CUDA Toolkit Version

  4. Search Tensorflow Version for CUDA and cuDNN

    • Version Python version Compiler Build tools cuDNN CUDA

    • tensorflow_gpu-2.10.0 3.7-3.10 MSVC 2019 Bazel 5.1.1 8.1 11.27

  5. Setup new miniconda environment tensorflow

    1
    $ conda create -n dl_lulc python=3.8

  6. Install cudatoolkit

    在我们的具体使用中，其实真正需要的并不是整个CUDA，而是cudatoolkit，所以我们在这里直接安装cudatoolkit，不需要再下载3个多G的CUDA来本地安装8。

    • Download cudatoolkit 11.2 from anaconda.org to local9.
    1
    2
    3
    4
    5
    $ conda activate dl_lulc

    $ conda install cudatoolkit=11.2 -c https://mirrors.bfsu.edu.cn/anaconda/pkgs/main/win-64/

    <!-- $ conda install cudatoolkit=11.2 -c https://mirrors.bfsu.edu.cn/anaconda/pkgs/main/win-64/ -->

  7. Install cudnn 8.1

    • Download cudnn 8.1 from anaconda.org to local10.

    • add conda-forge to channels

    1
    2
    3
    4
    5
    6
    7
    8
    conda config --add channels conda-forge

    conda config --set channel_priority strict

    conda search cudnn

    conda install cudnn=8.1 -c https://mirrors.bfsu.edu.cn/anaconda/pkgs/main/win-64/

  8. Install Tensorflow

    1
    conda install -c conda-forge tensorflow

  9. Install Tensorflow-gpu

    1
    pip install tensorflow-gpu==2.10.0

  10. Test sucessfully configed or not

    1
    2
    3
    4
    5
    $ python

    $ import tensorflow as tf

    $ tf.test.is_gpu_available()

    Sucessfully installed!

Python
#Deep Learning #LULC #Python #Machine Learning #Tensorflow #XML4LULC
Classification_Notes of XML4LULC Script
https://mengyuchi.gitlab.io/2023/02/28/Classification-Notes-of-XML4LULC-Script/
Author
Yuchi Meng
Posted on
February 28, 2023
Licensed under