JSON -> csv creating header line and padding header if found empty field2019 Community Moderator Electionadding an empty first line inside csv fileShell script to Capture the file name and size in csv file and add header on that filesplitting a CSV and keeping the header without intermediate filesDelete consecutive lines in CSV with duplicate values in one field, but keep the last lineextract values from a field in a line and append those values to this latter lineHow to remove duplicate lines in a CSV based on first field, and 1st n chars of 2nd field?Empty multiple .csv log files but retain the headerUsing AWK to add a new column with values to a csv file without creating empty new lines in between rowswriting files in a subdirectory to a csv file and save it to parent directory in linux command line

When should a starting writer get his own webpage?

How to test the sharpness of a knife?

How can a new country break out from a developed country without war?

DisplayForm problem with pi in FractionBox

Exit shell with shortcut (not typing exit) that closes session properly

Why is this tree refusing to shed its dead leaves?

Fair way to split coins

Exposing a company lying about themselves in a tightly knit industry: Is my career at risk on the long run?

Gauss brackets with double vertical lines

Writing in a Christian voice

How to balance a monster modification (zombie)?

Jem'Hadar, something strange about their life expectancy

Does the Shadow Magic sorcerer's Eyes of the Dark feature work on all Darkness spells or just his/her own?

Does fire aspect on a sword, destroy mob drops?

Print last inputted byte

Hot air balloons as primitive bombers

What kind of footwear is suitable for walking in micro gravity environment?

What will the Frenchman say?

Was World War I a war of liberals against authoritarians?

Nested Dynamic SOQL Query

Error in master's thesis, I do not know what to do

Animating wave motion in water

What are the rules for concealing thieves' tools (or items in general)?

Turning a hard to access nut?



JSON -> csv creating header line and padding header if found empty field



2019 Community Moderator Electionadding an empty first line inside csv fileShell script to Capture the file name and size in csv file and add header on that filesplitting a CSV and keeping the header without intermediate filesDelete consecutive lines in CSV with duplicate values in one field, but keep the last lineextract values from a field in a line and append those values to this latter lineHow to remove duplicate lines in a CSV based on first field, and 1st n chars of 2nd field?Empty multiple .csv log files but retain the headerUsing AWK to add a new column with values to a csv file without creating empty new lines in between rowswriting files in a subdirectory to a csv file and save it to parent directory in linux command line










2















I have a programme in bash that get JSONline files with several million of these object per line (See source)




"company_number": "09626947",
"data":
"address":
"address_line_1": "Troak Close",
"country": "England",
"locality": "Christchurch",
"postal_code": "BH23 3SR",
"premises": "9",
"region": "Dorset"
,
"country_of_residence": "United Kingdom",
"date_of_birth":
"month": 11,
"year": 1979
,
"etag": "7123fb76e4ad7ee7542da210a368baa4c89d5a06",
"kind": "individual-person-with-significant-control",
"links":
"self": "/company/09626947/persons-with-significant-control/individual/FFeqke7T3LvGvX6xmuGqi5SJXAk"
,
"name": "Ms Angela Lynette Miller",
"name_elements":
"forename": "Angela",
"middle_name": "Lynette",
"surname": "Miller",
"title": "Ms"
,
"nationality": "British",
"natures_of_control": [
"significant-influence-or-control"
],
"notified_on": "2016-06-06"




I have my JQ query that looks like this:



for file in psc_chunk_*; do
jq --slurp --raw-output 'def pad($n): range(0;$n) as $i |
.[$i]; ([.[] | .data.natures_of_control | length] | max) as $mx |
.[] |
select(.data) |
[.company_number, .data.kind, .data.address.address_line_1, .data.address.country, .data.address.locality, .data.address.postal_code, .data.address.premises, .data.identification.country_registered, .data.identification.legal_authority, .data.identification.legal_form, .data.identification.place_registered, .data.identification.registration_number, .data.ceased_on, .data.country_of_residence, "(.data.date_of_birth.year)-(.data.date_of_birth.month)", .data.etag, .data.links.self, .data.name, .data.name_elements.title, .data.name_elements.forename, .data.name_elements.middle_name, .data.name_elements.surname, .data.nationality, .data.notified_on, (.data.natures_of_control | pad($mx))] |
@csv' $file > $file.csv;
done


Which is probably hurting the eyes of many JQ pros out there - it is not efficient in extracting key:value pairs and if the provider happens to change name of a key my code wouldn't work anymore.



Is there a way to just flatten all the json into a csv keeping the keys as headers - with the extra difficulty that there is a list natures_of_control which has a varying number of entries (for which i used the pad function to get a rectangular result).










share|improve this question







New contributor




Tytire Recubans is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
























    2















    I have a programme in bash that get JSONline files with several million of these object per line (See source)




    "company_number": "09626947",
    "data":
    "address":
    "address_line_1": "Troak Close",
    "country": "England",
    "locality": "Christchurch",
    "postal_code": "BH23 3SR",
    "premises": "9",
    "region": "Dorset"
    ,
    "country_of_residence": "United Kingdom",
    "date_of_birth":
    "month": 11,
    "year": 1979
    ,
    "etag": "7123fb76e4ad7ee7542da210a368baa4c89d5a06",
    "kind": "individual-person-with-significant-control",
    "links":
    "self": "/company/09626947/persons-with-significant-control/individual/FFeqke7T3LvGvX6xmuGqi5SJXAk"
    ,
    "name": "Ms Angela Lynette Miller",
    "name_elements":
    "forename": "Angela",
    "middle_name": "Lynette",
    "surname": "Miller",
    "title": "Ms"
    ,
    "nationality": "British",
    "natures_of_control": [
    "significant-influence-or-control"
    ],
    "notified_on": "2016-06-06"




    I have my JQ query that looks like this:



    for file in psc_chunk_*; do
    jq --slurp --raw-output 'def pad($n): range(0;$n) as $i |
    .[$i]; ([.[] | .data.natures_of_control | length] | max) as $mx |
    .[] |
    select(.data) |
    [.company_number, .data.kind, .data.address.address_line_1, .data.address.country, .data.address.locality, .data.address.postal_code, .data.address.premises, .data.identification.country_registered, .data.identification.legal_authority, .data.identification.legal_form, .data.identification.place_registered, .data.identification.registration_number, .data.ceased_on, .data.country_of_residence, "(.data.date_of_birth.year)-(.data.date_of_birth.month)", .data.etag, .data.links.self, .data.name, .data.name_elements.title, .data.name_elements.forename, .data.name_elements.middle_name, .data.name_elements.surname, .data.nationality, .data.notified_on, (.data.natures_of_control | pad($mx))] |
    @csv' $file > $file.csv;
    done


    Which is probably hurting the eyes of many JQ pros out there - it is not efficient in extracting key:value pairs and if the provider happens to change name of a key my code wouldn't work anymore.



    Is there a way to just flatten all the json into a csv keeping the keys as headers - with the extra difficulty that there is a list natures_of_control which has a varying number of entries (for which i used the pad function to get a rectangular result).










    share|improve this question







    New contributor




    Tytire Recubans is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.






















      2












      2








      2








      I have a programme in bash that get JSONline files with several million of these object per line (See source)




      "company_number": "09626947",
      "data":
      "address":
      "address_line_1": "Troak Close",
      "country": "England",
      "locality": "Christchurch",
      "postal_code": "BH23 3SR",
      "premises": "9",
      "region": "Dorset"
      ,
      "country_of_residence": "United Kingdom",
      "date_of_birth":
      "month": 11,
      "year": 1979
      ,
      "etag": "7123fb76e4ad7ee7542da210a368baa4c89d5a06",
      "kind": "individual-person-with-significant-control",
      "links":
      "self": "/company/09626947/persons-with-significant-control/individual/FFeqke7T3LvGvX6xmuGqi5SJXAk"
      ,
      "name": "Ms Angela Lynette Miller",
      "name_elements":
      "forename": "Angela",
      "middle_name": "Lynette",
      "surname": "Miller",
      "title": "Ms"
      ,
      "nationality": "British",
      "natures_of_control": [
      "significant-influence-or-control"
      ],
      "notified_on": "2016-06-06"




      I have my JQ query that looks like this:



      for file in psc_chunk_*; do
      jq --slurp --raw-output 'def pad($n): range(0;$n) as $i |
      .[$i]; ([.[] | .data.natures_of_control | length] | max) as $mx |
      .[] |
      select(.data) |
      [.company_number, .data.kind, .data.address.address_line_1, .data.address.country, .data.address.locality, .data.address.postal_code, .data.address.premises, .data.identification.country_registered, .data.identification.legal_authority, .data.identification.legal_form, .data.identification.place_registered, .data.identification.registration_number, .data.ceased_on, .data.country_of_residence, "(.data.date_of_birth.year)-(.data.date_of_birth.month)", .data.etag, .data.links.self, .data.name, .data.name_elements.title, .data.name_elements.forename, .data.name_elements.middle_name, .data.name_elements.surname, .data.nationality, .data.notified_on, (.data.natures_of_control | pad($mx))] |
      @csv' $file > $file.csv;
      done


      Which is probably hurting the eyes of many JQ pros out there - it is not efficient in extracting key:value pairs and if the provider happens to change name of a key my code wouldn't work anymore.



      Is there a way to just flatten all the json into a csv keeping the keys as headers - with the extra difficulty that there is a list natures_of_control which has a varying number of entries (for which i used the pad function to get a rectangular result).










      share|improve this question







      New contributor




      Tytire Recubans is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.












      I have a programme in bash that get JSONline files with several million of these object per line (See source)




      "company_number": "09626947",
      "data":
      "address":
      "address_line_1": "Troak Close",
      "country": "England",
      "locality": "Christchurch",
      "postal_code": "BH23 3SR",
      "premises": "9",
      "region": "Dorset"
      ,
      "country_of_residence": "United Kingdom",
      "date_of_birth":
      "month": 11,
      "year": 1979
      ,
      "etag": "7123fb76e4ad7ee7542da210a368baa4c89d5a06",
      "kind": "individual-person-with-significant-control",
      "links":
      "self": "/company/09626947/persons-with-significant-control/individual/FFeqke7T3LvGvX6xmuGqi5SJXAk"
      ,
      "name": "Ms Angela Lynette Miller",
      "name_elements":
      "forename": "Angela",
      "middle_name": "Lynette",
      "surname": "Miller",
      "title": "Ms"
      ,
      "nationality": "British",
      "natures_of_control": [
      "significant-influence-or-control"
      ],
      "notified_on": "2016-06-06"




      I have my JQ query that looks like this:



      for file in psc_chunk_*; do
      jq --slurp --raw-output 'def pad($n): range(0;$n) as $i |
      .[$i]; ([.[] | .data.natures_of_control | length] | max) as $mx |
      .[] |
      select(.data) |
      [.company_number, .data.kind, .data.address.address_line_1, .data.address.country, .data.address.locality, .data.address.postal_code, .data.address.premises, .data.identification.country_registered, .data.identification.legal_authority, .data.identification.legal_form, .data.identification.place_registered, .data.identification.registration_number, .data.ceased_on, .data.country_of_residence, "(.data.date_of_birth.year)-(.data.date_of_birth.month)", .data.etag, .data.links.self, .data.name, .data.name_elements.title, .data.name_elements.forename, .data.name_elements.middle_name, .data.name_elements.surname, .data.nationality, .data.notified_on, (.data.natures_of_control | pad($mx))] |
      @csv' $file > $file.csv;
      done


      Which is probably hurting the eyes of many JQ pros out there - it is not efficient in extracting key:value pairs and if the provider happens to change name of a key my code wouldn't work anymore.



      Is there a way to just flatten all the json into a csv keeping the keys as headers - with the extra difficulty that there is a list natures_of_control which has a varying number of entries (for which i used the pad function to get a rectangular result).







      csv json jq






      share|improve this question







      New contributor




      Tytire Recubans is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.











      share|improve this question







      New contributor




      Tytire Recubans is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      share|improve this question




      share|improve this question






      New contributor




      Tytire Recubans is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      asked 12 hours ago









      Tytire RecubansTytire Recubans

      111




      111




      New contributor




      Tytire Recubans is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.





      New contributor





      Tytire Recubans is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.






      Tytire Recubans is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.




















          0






          active

          oldest

          votes











          Your Answer








          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "106"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: false,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          imageUploader:
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          ,
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );






          Tytire Recubans is a new contributor. Be nice, and check out our Code of Conduct.









          draft saved

          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f507005%2fjson-csv-creating-header-line-and-padding-header-if-found-empty-field%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown

























          0






          active

          oldest

          votes








          0






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes








          Tytire Recubans is a new contributor. Be nice, and check out our Code of Conduct.









          draft saved

          draft discarded


















          Tytire Recubans is a new contributor. Be nice, and check out our Code of Conduct.












          Tytire Recubans is a new contributor. Be nice, and check out our Code of Conduct.











          Tytire Recubans is a new contributor. Be nice, and check out our Code of Conduct.














          Thanks for contributing an answer to Unix & Linux Stack Exchange!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid


          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.

          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f507005%2fjson-csv-creating-header-line-and-padding-header-if-found-empty-field%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          getting Checkpoint VPN SSL Network Extender working in the command lineHow to connect to CheckPoint VPN on Ubuntu 18.04LTS?Will the Linux ( red-hat ) Open VPNC Client connect to checkpoint or nortel VPN gateways?VPN client for linux machine + support checkpoint gatewayVPN SSL Network Extender in FirefoxLinux Checkpoint SNX tool configuration issuesCheck Point - Connect under Linux - snx + OTPSNX VPN Ububuntu 18.XXUsing Checkpoint VPN SSL Network Extender CLI with certificateVPN with network manager (nm-applet) is not workingWill the Linux ( red-hat ) Open VPNC Client connect to checkpoint or nortel VPN gateways?VPN client for linux machine + support checkpoint gatewayImport VPN config files to NetworkManager from command lineTrouble connecting to VPN using network-manager, while command line worksStart a VPN connection with PPTP protocol on command linestarting a docker service daemon breaks the vpn networkCan't connect to vpn with Network-managerVPN SSL Network Extender in FirefoxUsing Checkpoint VPN SSL Network Extender CLI with certificate

          Cannot Extend partition with GParted The 2019 Stack Overflow Developer Survey Results Are In Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern) 2019 Community Moderator Election ResultsCan't increase partition size with GParted?GParted doesn't recognize the unallocated space after my current partitionWhat is the best way to add unallocated space located before to Ubuntu 12.04 partition with GParted live?I can't figure out how to extend my Arch home partition into free spaceGparted Linux Mint 18.1 issueTrying to extend but swap partition is showing as Unknown in Gparted, shows proper from fdiskRearrange partitions in gparted to extend a partitionUnable to extend partition even though unallocated space is next to it using GPartedAllocate free space to root partitiongparted: how to merge unallocated space with a partition

          Marilyn Monroe Ny fiainany manokana | Jereo koa | Meny fitetezanafanitarana azy.