Data prepration for logistic regression : Value either “not available” or a “year”Logistic Regression implementation does not convergeLogistic regression on biased dataIntuition for Logistic Regression PerformanceUpdate statement for Logistic RegressionSimple logistic regression wrong predictionslogistic regressionThe test of randomness was a logistic regression predicting missingness from all other variablesDealing with NaN (missing) values for Logistic Regression- Best practices?Logistic regression in pythonWhy is logistic regression not sigmoidal?
Would this string work as string?
python displays `n` instead of breaking a line
What is the tangent at a sharp point on a curve?
Mac Mini (2018) 10Gb Port Compatibility
I keep switching characters, how do I stop?
Why do Radio Buttons not fill the entire outer circle?
If the Dominion rule using their Jem'Hadar troops, why is their life expectancy so low?
Derivative of an interpolated function
Asserting that Atheism and Theism are both faith based positions
What (if any) is the reason to buy in small local stores?
Why is indicated airspeed rather than ground speed used during the takeoff roll?
Travelling in US for more than 90 days
How to preserve electronics (computers, ipads, phones) for hundreds of years?
Started in 1987 vs. Starting in 1987
How would a solely written language work mechanically
Can you take a "free object interaction" while incapacitated?
Put the phone down / Put down the phone
How do I lift the insulation blower into the attic?
Pre-Employment Background Check With Consent For Future Checks
Not hide and seek
What 1968 Moog synthesizer was used in the Movie Apollo 11?
How can a new country break out from a developed country without war?
What is the meaning of "You've never met a graph you didn't like?"
Why can't I get pgrep output right to variable on bash script?
Data prepration for logistic regression : Value either “not available” or a “year”
Logistic Regression implementation does not convergeLogistic regression on biased dataIntuition for Logistic Regression PerformanceUpdate statement for Logistic RegressionSimple logistic regression wrong predictionslogistic regressionThe test of randomness was a logistic regression predicting missingness from all other variablesDealing with NaN (missing) values for Logistic Regression- Best practices?Logistic regression in pythonWhy is logistic regression not sigmoidal?
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked 18 hours ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked 18 hours ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
17 hours ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
15 hours ago
add a comment |
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked 18 hours ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
logistic-regression missing-data
asked 18 hours ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 18 hours ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 18 hours ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 18 hours ago
toomtoom
1112
asked 18 hours ago
toomtoom
1112
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
17 hours ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
15 hours ago
add a comment |
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
17 hours ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
15 hours ago
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
17 hours ago
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
17 hours ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
15 hours ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
15 hours ago
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 17 hours ago
EsmailianEsmailian
1,536113
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
15 hours ago
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
toom is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47586%2fdata-prepration-for-logistic-regression-value-either-not-available-or-a-yea%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 17 hours ago
EsmailianEsmailian
1,536113
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
15 hours ago
add a comment |
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 17 hours ago
EsmailianEsmailian
1,536113
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
15 hours ago
add a comment |
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 17 hours ago
EsmailianEsmailian
1,536113
$endgroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 17 hours ago
EsmailianEsmailian
1,536113
edited 12 hours ago
answered 17 hours ago
EsmailianEsmailian
1,536113
answered 17 hours ago
EsmailianEsmailian
1,536113
answered 17 hours ago
EsmailianEsmailian
1,536113
1,536113
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
15 hours ago
add a comment |
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
15 hours ago
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
15 hours ago
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
15 hours ago
add a comment |
toom is a new contributor. Be nice, and check out our Code of Conduct.
toom is a new contributor. Be nice, and check out our Code of Conduct.
toom is a new contributor. Be nice, and check out our Code of Conduct.
toom is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47586%2fdata-prepration-for-logistic-regression-value-either-not-available-or-a-yea%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
17 hours ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
15 hours ago