JSON -> csv creating header line and padding header if found empty field2019 Community Moderator Electionadding an empty first line inside csv fileShell script to Capture the file name and size in csv file and add header on that filesplitting a CSV and keeping the header without intermediate filesDelete consecutive lines in CSV with duplicate values in one field, but keep the last lineextract values from a field in a line and append those values to this latter lineHow to remove duplicate lines in a CSV based on first field, and 1st n chars of 2nd field?Empty multiple .csv log files but retain the headerUsing AWK to add a new column with values to a csv file without creating empty new lines in between rowswriting files in a subdirectory to a csv file and save it to parent directory in linux command line
When should a starting writer get his own webpage?
How to test the sharpness of a knife?
How can a new country break out from a developed country without war?
DisplayForm problem with pi in FractionBox
Exit shell with shortcut (not typing exit) that closes session properly
Why is this tree refusing to shed its dead leaves?
Fair way to split coins
Exposing a company lying about themselves in a tightly knit industry: Is my career at risk on the long run?
Gauss brackets with double vertical lines
Writing in a Christian voice
How to balance a monster modification (zombie)?
Jem'Hadar, something strange about their life expectancy
Does the Shadow Magic sorcerer's Eyes of the Dark feature work on all Darkness spells or just his/her own?
Does fire aspect on a sword, destroy mob drops?
Print last inputted byte
Hot air balloons as primitive bombers
What kind of footwear is suitable for walking in micro gravity environment?
What will the Frenchman say?
Was World War I a war of liberals against authoritarians?
Nested Dynamic SOQL Query
Error in master's thesis, I do not know what to do
Animating wave motion in water
What are the rules for concealing thieves' tools (or items in general)?
Turning a hard to access nut?
JSON -> csv creating header line and padding header if found empty field
2019 Community Moderator Electionadding an empty first line inside csv fileShell script to Capture the file name and size in csv file and add header on that filesplitting a CSV and keeping the header without intermediate filesDelete consecutive lines in CSV with duplicate values in one field, but keep the last lineextract values from a field in a line and append those values to this latter lineHow to remove duplicate lines in a CSV based on first field, and 1st n chars of 2nd field?Empty multiple .csv log files but retain the headerUsing AWK to add a new column with values to a csv file without creating empty new lines in between rowswriting files in a subdirectory to a csv file and save it to parent directory in linux command line
I have a programme in bash that get JSONline files with several million of these object per line (See source)
"company_number": "09626947",
"data":
"address":
"address_line_1": "Troak Close",
"country": "England",
"locality": "Christchurch",
"postal_code": "BH23 3SR",
"premises": "9",
"region": "Dorset"
,
"country_of_residence": "United Kingdom",
"date_of_birth":
"month": 11,
"year": 1979
,
"etag": "7123fb76e4ad7ee7542da210a368baa4c89d5a06",
"kind": "individual-person-with-significant-control",
"links":
"self": "/company/09626947/persons-with-significant-control/individual/FFeqke7T3LvGvX6xmuGqi5SJXAk"
,
"name": "Ms Angela Lynette Miller",
"name_elements":
"forename": "Angela",
"middle_name": "Lynette",
"surname": "Miller",
"title": "Ms"
,
"nationality": "British",
"natures_of_control": [
"significant-influence-or-control"
],
"notified_on": "2016-06-06"
I have my JQ query that looks like this:
for file in psc_chunk_*; do
jq --slurp --raw-output 'def pad($n): range(0;$n) as $i |
.[$i]; ([.[] | .data.natures_of_control | length] | max) as $mx |
.[] |
select(.data) |
[.company_number, .data.kind, .data.address.address_line_1, .data.address.country, .data.address.locality, .data.address.postal_code, .data.address.premises, .data.identification.country_registered, .data.identification.legal_authority, .data.identification.legal_form, .data.identification.place_registered, .data.identification.registration_number, .data.ceased_on, .data.country_of_residence, "(.data.date_of_birth.year)-(.data.date_of_birth.month)", .data.etag, .data.links.self, .data.name, .data.name_elements.title, .data.name_elements.forename, .data.name_elements.middle_name, .data.name_elements.surname, .data.nationality, .data.notified_on, (.data.natures_of_control | pad($mx))] |
@csv' $file > $file.csv;
done
Which is probably hurting the eyes of many JQ pros out there - it is not efficient in extracting key:value pairs and if the provider happens to change name of a key my code wouldn't work anymore.
Is there a way to just flatten all the json into a csv keeping the keys as headers - with the extra difficulty that there is a list natures_of_control
which has a varying number of entries (for which i used the pad function to get a rectangular result).
csv json jq
New contributor
add a comment |
I have a programme in bash that get JSONline files with several million of these object per line (See source)
"company_number": "09626947",
"data":
"address":
"address_line_1": "Troak Close",
"country": "England",
"locality": "Christchurch",
"postal_code": "BH23 3SR",
"premises": "9",
"region": "Dorset"
,
"country_of_residence": "United Kingdom",
"date_of_birth":
"month": 11,
"year": 1979
,
"etag": "7123fb76e4ad7ee7542da210a368baa4c89d5a06",
"kind": "individual-person-with-significant-control",
"links":
"self": "/company/09626947/persons-with-significant-control/individual/FFeqke7T3LvGvX6xmuGqi5SJXAk"
,
"name": "Ms Angela Lynette Miller",
"name_elements":
"forename": "Angela",
"middle_name": "Lynette",
"surname": "Miller",
"title": "Ms"
,
"nationality": "British",
"natures_of_control": [
"significant-influence-or-control"
],
"notified_on": "2016-06-06"
I have my JQ query that looks like this:
for file in psc_chunk_*; do
jq --slurp --raw-output 'def pad($n): range(0;$n) as $i |
.[$i]; ([.[] | .data.natures_of_control | length] | max) as $mx |
.[] |
select(.data) |
[.company_number, .data.kind, .data.address.address_line_1, .data.address.country, .data.address.locality, .data.address.postal_code, .data.address.premises, .data.identification.country_registered, .data.identification.legal_authority, .data.identification.legal_form, .data.identification.place_registered, .data.identification.registration_number, .data.ceased_on, .data.country_of_residence, "(.data.date_of_birth.year)-(.data.date_of_birth.month)", .data.etag, .data.links.self, .data.name, .data.name_elements.title, .data.name_elements.forename, .data.name_elements.middle_name, .data.name_elements.surname, .data.nationality, .data.notified_on, (.data.natures_of_control | pad($mx))] |
@csv' $file > $file.csv;
done
Which is probably hurting the eyes of many JQ pros out there - it is not efficient in extracting key:value pairs and if the provider happens to change name of a key my code wouldn't work anymore.
Is there a way to just flatten all the json into a csv keeping the keys as headers - with the extra difficulty that there is a list natures_of_control
which has a varying number of entries (for which i used the pad function to get a rectangular result).
csv json jq
New contributor
add a comment |
I have a programme in bash that get JSONline files with several million of these object per line (See source)
"company_number": "09626947",
"data":
"address":
"address_line_1": "Troak Close",
"country": "England",
"locality": "Christchurch",
"postal_code": "BH23 3SR",
"premises": "9",
"region": "Dorset"
,
"country_of_residence": "United Kingdom",
"date_of_birth":
"month": 11,
"year": 1979
,
"etag": "7123fb76e4ad7ee7542da210a368baa4c89d5a06",
"kind": "individual-person-with-significant-control",
"links":
"self": "/company/09626947/persons-with-significant-control/individual/FFeqke7T3LvGvX6xmuGqi5SJXAk"
,
"name": "Ms Angela Lynette Miller",
"name_elements":
"forename": "Angela",
"middle_name": "Lynette",
"surname": "Miller",
"title": "Ms"
,
"nationality": "British",
"natures_of_control": [
"significant-influence-or-control"
],
"notified_on": "2016-06-06"
I have my JQ query that looks like this:
for file in psc_chunk_*; do
jq --slurp --raw-output 'def pad($n): range(0;$n) as $i |
.[$i]; ([.[] | .data.natures_of_control | length] | max) as $mx |
.[] |
select(.data) |
[.company_number, .data.kind, .data.address.address_line_1, .data.address.country, .data.address.locality, .data.address.postal_code, .data.address.premises, .data.identification.country_registered, .data.identification.legal_authority, .data.identification.legal_form, .data.identification.place_registered, .data.identification.registration_number, .data.ceased_on, .data.country_of_residence, "(.data.date_of_birth.year)-(.data.date_of_birth.month)", .data.etag, .data.links.self, .data.name, .data.name_elements.title, .data.name_elements.forename, .data.name_elements.middle_name, .data.name_elements.surname, .data.nationality, .data.notified_on, (.data.natures_of_control | pad($mx))] |
@csv' $file > $file.csv;
done
Which is probably hurting the eyes of many JQ pros out there - it is not efficient in extracting key:value pairs and if the provider happens to change name of a key my code wouldn't work anymore.
Is there a way to just flatten all the json into a csv keeping the keys as headers - with the extra difficulty that there is a list natures_of_control
which has a varying number of entries (for which i used the pad function to get a rectangular result).
csv json jq
New contributor
I have a programme in bash that get JSONline files with several million of these object per line (See source)
"company_number": "09626947",
"data":
"address":
"address_line_1": "Troak Close",
"country": "England",
"locality": "Christchurch",
"postal_code": "BH23 3SR",
"premises": "9",
"region": "Dorset"
,
"country_of_residence": "United Kingdom",
"date_of_birth":
"month": 11,
"year": 1979
,
"etag": "7123fb76e4ad7ee7542da210a368baa4c89d5a06",
"kind": "individual-person-with-significant-control",
"links":
"self": "/company/09626947/persons-with-significant-control/individual/FFeqke7T3LvGvX6xmuGqi5SJXAk"
,
"name": "Ms Angela Lynette Miller",
"name_elements":
"forename": "Angela",
"middle_name": "Lynette",
"surname": "Miller",
"title": "Ms"
,
"nationality": "British",
"natures_of_control": [
"significant-influence-or-control"
],
"notified_on": "2016-06-06"
I have my JQ query that looks like this:
for file in psc_chunk_*; do
jq --slurp --raw-output 'def pad($n): range(0;$n) as $i |
.[$i]; ([.[] | .data.natures_of_control | length] | max) as $mx |
.[] |
select(.data) |
[.company_number, .data.kind, .data.address.address_line_1, .data.address.country, .data.address.locality, .data.address.postal_code, .data.address.premises, .data.identification.country_registered, .data.identification.legal_authority, .data.identification.legal_form, .data.identification.place_registered, .data.identification.registration_number, .data.ceased_on, .data.country_of_residence, "(.data.date_of_birth.year)-(.data.date_of_birth.month)", .data.etag, .data.links.self, .data.name, .data.name_elements.title, .data.name_elements.forename, .data.name_elements.middle_name, .data.name_elements.surname, .data.nationality, .data.notified_on, (.data.natures_of_control | pad($mx))] |
@csv' $file > $file.csv;
done
Which is probably hurting the eyes of many JQ pros out there - it is not efficient in extracting key:value pairs and if the provider happens to change name of a key my code wouldn't work anymore.
Is there a way to just flatten all the json into a csv keeping the keys as headers - with the extra difficulty that there is a list natures_of_control
which has a varying number of entries (for which i used the pad function to get a rectangular result).
csv json jq
csv json jq
New contributor
New contributor
New contributor
asked 12 hours ago
Tytire RecubansTytire Recubans
111
111
New contributor
New contributor
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Tytire Recubans is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f507005%2fjson-csv-creating-header-line-and-padding-header-if-found-empty-field%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Tytire Recubans is a new contributor. Be nice, and check out our Code of Conduct.
Tytire Recubans is a new contributor. Be nice, and check out our Code of Conduct.
Tytire Recubans is a new contributor. Be nice, and check out our Code of Conduct.
Tytire Recubans is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f507005%2fjson-csv-creating-header-line-and-padding-header-if-found-empty-field%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown