Skip to content

Denoising

Denoising

EWMAFilter

Bases: TransformerStep

Filter each feature using EWM (Exponential Moving Window)

Parameters:

Name Type Description Default
span float

Time constant of the EMA (Exponential Moving Average)

 required
name Optional[str]

Name of the step, by default None

 None
Source code in ceruleo/transformation/features/denoising.py 
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
class EWMAFilter(TransformerStep):
    """
    Filter each feature using EWM (Exponential Moving Window)

    Parameters:
        span: Time constant of the EMA (Exponential Moving Average)
        name: Name of the step, by default None
    """

    def __init__(self, span: float, name: Optional[str] = None):
        super().__init__(name=name)
        self.span = span

    def transform(self, X: pd.DataFrame, y=None) -> pd.DataFrame:
        """
        Return a new dataframe with the features filtered

        Parameters:
            X: Input life

        Returns:
            A new DatafFrame with the same index as the input with the features filtered
        """
        return X.ewm(span=self.span, ignore_na=True).mean()

transform(X, y=None)

Return a new dataframe with the features filtered

Parameters:

Name Type Description Default
X DataFrame

Input life

 required

Returns:

Type Description
DataFrame

A new DatafFrame with the same index as the input with the features filtered
Source code in ceruleo/transformation/features/denoising.py 
214
215
216
217
218
219
220
221
222
223
224
def transform(self, X: pd.DataFrame, y=None) -> pd.DataFrame:
    """
    Return a new dataframe with the features filtered

    Parameters:
        X: Input life

    Returns:
        A new DatafFrame with the same index as the input with the features filtered
    """
    return X.ewm(span=self.span, ignore_na=True).mean()

GaussianFilter

Bases: TransformerStep

Apply a gaussian filter

Parameters:

Name Type Description Default
window_size int

Size of the gaussian filter

 required
std float

Standard deviation of the filter

 required
min_points int

Minimun nomber of points of the rolling window, by default 1

 1
center bool

Wether the guassian window should be centered, by default False

 False
Source code in ceruleo/transformation/features/denoising.py 
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
class GaussianFilter(TransformerStep):
    """
    Apply a gaussian filter

    Parameters:
        window_size: Size of the gaussian filter
        std: Standard deviation of the filter
        min_points: Minimun nomber of points of the rolling window, by default 1
        center: Wether the guassian window should be centered, by default False
    """

    def __init__(
        self,
        window_size: int,
        std: float,
        min_points: int = 1,
        center: bool = False,
        *args,
        **kwargs,
    ):
        super().__init__(**kwargs)
        self.window_size = window_size
        self.std = std
        self.center = center
        self.min_points = min_points

    def transform(self, X: pd.DataFrame) -> pd.DataFrame:
        """
        Return a new dataframe with the features filtered

        Parameters:
            X: Input life

        Returns:
            A new DatafFrame with the same index as the input with the features filtered
        """
        return X.rolling(
            window=self.window_size,
            win_type="gaussian",
            center=self.center,
            min_periods=self.min_points,
        ).mean(std=self.std)

transform(X)

Return a new dataframe with the features filtered

Parameters:

Name Type Description Default
X DataFrame

Input life

 required

Returns:

Type Description
DataFrame

A new DatafFrame with the same index as the input with the features filtered
Source code in ceruleo/transformation/features/denoising.py 
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
def transform(self, X: pd.DataFrame) -> pd.DataFrame:
    """
    Return a new dataframe with the features filtered

    Parameters:
        X: Input life

    Returns:
        A new DatafFrame with the same index as the input with the features filtered
    """
    return X.rolling(
        window=self.window_size,
        win_type="gaussian",
        center=self.center,
        min_periods=self.min_points,
    ).mean(std=self.std)

MeanFilter

Bases: TransformerStep

Filter each feature using a rolling mean filter

Parameters:

Name Type Description Default
window int

Size of the rolling window

 required
min_periods int

Minimum number of non-null points of the rolling window, by default 15

 15
name Optional[str]

Name of the step, by default None

 None
center

Wether the guassian window should be centered, by default False

 True
Source code in ceruleo/transformation/features/denoising.py 
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
class MeanFilter(TransformerStep):
    """
    Filter each feature using a rolling mean filter

    Parameters:
        window: Size of the rolling window
        min_periods: Minimum number of non-null points of the rolling window, by default 15
        name: Name of the step, by default None
        center: Wether the guassian window should be centered, by default False
    """

    def __init__(
        self,
        window: int,
        center=True,
        min_periods: int = 15,
        name: Optional[str] = None,
    ):
        super().__init__(name=name)
        self.window = window
        self.min_periods = min_periods
        self.center = center

    def transform(self, X: pd.DataFrame, y=None) -> pd.DataFrame:
        """
        Return a new dataframe with the features filtered

        Parameters:
            X: Input life

        Returns:
            A new DatafFrame with the same index as the input with the features filtered
        """
        return X.rolling(
            self.window,
            min_periods=self.min_periods,
            center=self.center,
        ).mean(numeric_only=True)

transform(X, y=None)

Return a new dataframe with the features filtered

Parameters:

Name Type Description Default
X DataFrame

Input life

 required

Returns:

Type Description
DataFrame

A new DatafFrame with the same index as the input with the features filtered
Source code in ceruleo/transformation/features/denoising.py 
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
def transform(self, X: pd.DataFrame, y=None) -> pd.DataFrame:
    """
    Return a new dataframe with the features filtered

    Parameters:
        X: Input life

    Returns:
        A new DatafFrame with the same index as the input with the features filtered
    """
    return X.rolling(
        self.window,
        min_periods=self.min_periods,
        center=self.center,
    ).mean(numeric_only=True)

MedianFilter

Bases: TransformerStep

Filter each feature using a rolling median filter

Parameters:

Name Type Description Default
window int

Size of the rolling window

 required
min_periods int

Minimum number of points of the rolling window, by default 15

 15
name Optional[str]

Name of the step, by default None

 None
Source code in ceruleo/transformation/features/denoising.py 
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
class MedianFilter(TransformerStep):
    """Filter each feature using a rolling median filter

    Parameters:
        window: Size of the rolling window
        min_periods: Minimum number of points of the rolling window, by default 15
        name: Name of the step, by default None
    """

    def __init__(self, window: int, min_periods: int = 15, name: Optional[str] = None):
        super().__init__(name=name)
        self.window = window
        self.min_periods = min_periods

    def transform(self, X: pd.DataFrame, y=None) -> pd.DataFrame:
        """
        Return a new dataframe with the features filtered

        Parameters:
            X: Input life

        Returns:
            A new DatafFrame with the same index as the input with the features filtered
        """
        return X.rolling(self.window, min_periods=self.min_periods).median(
            numeric_only=True
        )

transform(X, y=None)

Return a new dataframe with the features filtered

Parameters:

Name Type Description Default
X DataFrame

Input life

 required

Returns:

Type Description
DataFrame

A new DatafFrame with the same index as the input with the features filtered
Source code in ceruleo/transformation/features/denoising.py 
 98
 99
100
101
102
103
104
105
106
107
108
109
110
def transform(self, X: pd.DataFrame, y=None) -> pd.DataFrame:
    """
    Return a new dataframe with the features filtered

    Parameters:
        X: Input life

    Returns:
        A new DatafFrame with the same index as the input with the features filtered
    """
    return X.rolling(self.window, min_periods=self.min_periods).median(
        numeric_only=True
    )

MultiDimensionalKMeans

Bases: TransformerStep

Clusterize data points and replace each feature with the centroid feature it belongs to

Parameters:

Name Type Description Default
n_clusters int

Number of clusters to obtain by default 5

 5
name Optional[str]

Name of the step, by default None

 None
Source code in ceruleo/transformation/features/denoising.py 
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
class MultiDimensionalKMeans(TransformerStep):
    """
    Clusterize data points and replace each feature with the centroid feature it belongs to

    Parameters:
        n_clusters: Number of clusters to obtain by default 5
        name: Name of the step, by default None
    """

    def __init__(self, n_clusters: int = 5, name: Optional[str] = None):
        super().__init__(name=name)
        self.n_clusters = n_clusters
        self.clusters = MiniBatchKMeans(n_clusters=self.n_clusters, n_init="auto")

    def partial_fit(self, X: pd.DataFrame):
        """
        Fit the model to the input data to obtain the clusters

        Parameters:
            X: Input life
        """
        self.clusters.partial_fit(X)
        return self

    def transform(self, X: pd.DataFrame, y=None) -> pd.DataFrame:
        """Transform the input life with the centroid information

        Parameters:
            X: Input life

        Returns:
            A new DataFrame in which each point was replaced by the centroid it belongs to
        """

        X = X.copy()
        X[:] = self.clusters.cluster_centers_[self.clusters.predict(X)]
        return X

partial_fit(X)

Fit the model to the input data to obtain the clusters

Parameters:

Name Type Description Default
X DataFrame

Input life

 required
Source code in ceruleo/transformation/features/denoising.py 
176
177
178
179
180
181
182
183
184
def partial_fit(self, X: pd.DataFrame):
    """
    Fit the model to the input data to obtain the clusters

    Parameters:
        X: Input life
    """
    self.clusters.partial_fit(X)
    return self

transform(X, y=None)

Transform the input life with the centroid information

Parameters:

Name Type Description Default
X DataFrame

Input life

 required

Returns:

Type Description
DataFrame

A new DataFrame in which each point was replaced by the centroid it belongs to
Source code in ceruleo/transformation/features/denoising.py 
186
187
188
189
190
191
192
193
194
195
196
197
198
def transform(self, X: pd.DataFrame, y=None) -> pd.DataFrame:
    """Transform the input life with the centroid information

    Parameters:
        X: Input life

    Returns:
        A new DataFrame in which each point was replaced by the centroid it belongs to
    """

    X = X.copy()
    X[:] = self.clusters.cluster_centers_[self.clusters.predict(X)]
    return X

OneDimensionalKMeans

Bases: TransformerStep

Clusterize each feature into a number of clusters

Parameters:

Name Type Description Default
n_clusters int

Number of clusters, by default 5

 5
Source code in ceruleo/transformation/features/denoising.py 
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
class OneDimensionalKMeans(TransformerStep):
    """
    Clusterize each feature into a number of clusters

    Parameters:
        n_clusters: Number of clusters, by default 5

    """

    def __init__(self, n_clusters: int = 5, name: Optional[str] = None):
        super().__init__(name=name)
        self.clusters = {}
        self.n_clusters = n_clusters

    def partial_fit(self, X: pd.DataFrame):
        """
        Fit the model to the input data to obtain the clusters

        Parameters:
            X: Input life
        """
        if len(self.clusters) == 0:
            for c in X.columns:
                self.clusters[c] = MiniBatchKMeans(
                    n_clusters=self.n_clusters, n_init="auto"
                )

        for c in X.columns:
            self.clusters[c].partial_fit(np.atleast_2d(X[c]).T)
        return self

    def transform(self, X: pd.DataFrame, y=None) -> pd.DataFrame:
        """
        Transform the input dataframe

        Parameters:
            X: Input life

        Returns:
            A new DataFrame with the same index as the input. Each feature is replaced with the clusters of each point
        """
        X = X.copy()
        for c in X.columns:
            X[c] = self.clusters[c].cluster_centers_[
                self.clusters[c].predict(np.atleast_2d(X[c]).T)
            ]
        return X

partial_fit(X)

Fit the model to the input data to obtain the clusters

Parameters:

Name Type Description Default
X DataFrame

Input life

 required
Source code in ceruleo/transformation/features/denoising.py 
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
def partial_fit(self, X: pd.DataFrame):
    """
    Fit the model to the input data to obtain the clusters

    Parameters:
        X: Input life
    """
    if len(self.clusters) == 0:
        for c in X.columns:
            self.clusters[c] = MiniBatchKMeans(
                n_clusters=self.n_clusters, n_init="auto"
            )

    for c in X.columns:
        self.clusters[c].partial_fit(np.atleast_2d(X[c]).T)
    return self

transform(X, y=None)

Transform the input dataframe

Parameters:

Name Type Description Default
X DataFrame

Input life

 required

Returns:

Type Description
DataFrame

A new DataFrame with the same index as the input. Each feature is replaced with the clusters of each point
Source code in ceruleo/transformation/features/denoising.py 
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
def transform(self, X: pd.DataFrame, y=None) -> pd.DataFrame:
    """
    Transform the input dataframe

    Parameters:
        X: Input life

    Returns:
        A new DataFrame with the same index as the input. Each feature is replaced with the clusters of each point
    """
    X = X.copy()
    for c in X.columns:
        X[c] = self.clusters[c].cluster_centers_[
            self.clusters[c].predict(np.atleast_2d(X[c]).T)
        ]
    return X

SavitzkyGolayTransformer

Bases: TransformerStep

Filter each feature using LOESS

Parameters:

Name Type Description Default
window int

window size of the filter

 required
order int

Order of the filter, by default 2

 2
name Optional[str]

Step name

 None
Source code in ceruleo/transformation/features/denoising.py 
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
class SavitzkyGolayTransformer(TransformerStep):
    """Filter each feature using LOESS

    Parameters:
        window: window size of the filter
        order:  Order of the filter, by default 2
        name: Step name
    """

    def __init__(self, window: int, order: int = 2, name: Optional[str] = None):
        super().__init__(name=name)
        self.window = window
        self.order = order

    def transform(self, X: pd.DataFrame, y=None) -> pd.DataFrame:
        """
        Return a new dataframe with the features filtered

        Parameters:
            X: Input life

        Returns:
            A new DatafFrame with the same index as the input with the features filtered
        """
        if X.shape[0] > self.window:
            return pd.DataFrame(
                savgol_filter(X, self.window, self.order, axis=0),
                columns=X.columns,
                index=X.index,
            )
        else:
            return X

transform(X, y=None)

Return a new dataframe with the features filtered

Parameters:

Name Type Description Default
X DataFrame

Input life

 required

Returns:

Type Description
DataFrame

A new DatafFrame with the same index as the input with the features filtered
Source code in ceruleo/transformation/features/denoising.py 
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
def transform(self, X: pd.DataFrame, y=None) -> pd.DataFrame:
    """
    Return a new dataframe with the features filtered

    Parameters:
        X: Input life

    Returns:
        A new DatafFrame with the same index as the input with the features filtered
    """
    if X.shape[0] > self.window:
        return pd.DataFrame(
            savgol_filter(X, self.window, self.order, axis=0),
            columns=X.columns,
            index=X.index,
        )
    else:
        return X