What is a good way to store processed CSV data to train model in Python?2019 Community Moderator Electionpython - Will this data mining approach work? Is it a good idea?Tools to perform SQL analytics on 350TB of csv dataimporting csv data in pythonCreating data model out of .csv file using PythonHow to store strings in CSV with new line characters?How to properly save and load an intermediate model in Keras?How can I merge 2+ DataFrame objects without duplicating column names?How to handle preprocessing (StandardScaler, LabelEncoder) when using data generator to train?Repeated groups of columns in data analysisEfficiently training big models on big dataframes with big samples, with crossvalidation and shuffling, and limited ram

Hostile work environment after whistle-blowing on coworker and our boss. What do I do?

How can I kill an app using Terminal?

Large drywall patch supports

I'm in charge of equipment buying but no one's ever happy with what I choose. How to fix this?

How does buying out courses with grant money work?

Different result between scanning in Epson's "color negative film" mode and scanning in positive -> invert curve in post?

Why are there no referendums in the US?

Can the discrete variable be a negative number?

Unreliable Magic - Is it worth it?

Why escape if the_content isnt?

How to pronounce the slash sign

Valid Badminton Score?

How does it work when somebody invests in my business?

Go Pregnant or Go Home

Roman Numeral Treatment of Suspensions

How can a function with a hole (removable discontinuity) equal a function with no hole?

Tiptoe or tiphoof? Adjusting words to better fit fantasy races

How do I rename a Linux host without needing to reboot for the rename to take effect?

Is the destination of a commercial flight important for the pilot?

Is `x >> pure y` equivalent to `liftM (const y) x`

How do scammers retract money, while you can’t?

What can we do to stop prior company from asking us questions?

Method to test if a number is a perfect power?

What does 算不上 mean in 算不上太美好的日子?



What is a good way to store processed CSV data to train model in Python?



2019 Community Moderator Electionpython - Will this data mining approach work? Is it a good idea?Tools to perform SQL analytics on 350TB of csv dataimporting csv data in pythonCreating data model out of .csv file using PythonHow to store strings in CSV with new line characters?How to properly save and load an intermediate model in Keras?How can I merge 2+ DataFrame objects without duplicating column names?How to handle preprocessing (StandardScaler, LabelEncoder) when using data generator to train?Repeated groups of columns in data analysisEfficiently training big models on big dataframes with big samples, with crossvalidation and shuffling, and limited ram










1












$begingroup$


I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.



Some options appear to be:



  • HDFS

  • HDF5

  • HDFS3

  • PyArrow





python keras dataset csv serialisation







share|improve this question











$endgroup$











  • $begingroup$
    When I want to got 5 mts in distance, I would rather walk than to take a car.
    $endgroup$
    – Kiritee Gak
    yesterday










  • $begingroup$
    I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast.
    $endgroup$
    – honar.cs
    yesterday






  • 1




    $begingroup$
    Just leave it as CSV you don't need to do anything
    $endgroup$
    – arhwerhwe
    yesterday






  • 1




    $begingroup$
    Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ...
    $endgroup$
    – n1tk
    yesterday















1












$begingroup$


I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.



Some options appear to be:



  • HDFS

  • HDF5

  • HDFS3

  • PyArrow





python keras dataset csv serialisation







share|improve this question











$endgroup$











  • $begingroup$
    When I want to got 5 mts in distance, I would rather walk than to take a car.
    $endgroup$
    – Kiritee Gak
    yesterday










  • $begingroup$
    I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast.
    $endgroup$
    – honar.cs
    yesterday






  • 1




    $begingroup$
    Just leave it as CSV you don't need to do anything
    $endgroup$
    – arhwerhwe
    yesterday






  • 1




    $begingroup$
    Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ...
    $endgroup$
    – n1tk
    yesterday













1












1








1





$begingroup$


I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.



Some options appear to be:



  • HDFS

  • HDF5

  • HDFS3

  • PyArrow





python keras dataset csv serialisation







share|improve this question











$endgroup$




I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.



Some options appear to be:



  • HDFS

  • HDF5

  • HDFS3

  • PyArrow





python keras dataset csv serialisation




python keras dataset csv serialisation






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited yesterday









Media

7,42262163




7,42262163










asked yesterday









B SevenB Seven

21218




21218











  • $begingroup$
    When I want to got 5 mts in distance, I would rather walk than to take a car.
    $endgroup$
    – Kiritee Gak
    yesterday










  • $begingroup$
    I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast.
    $endgroup$
    – honar.cs
    yesterday






  • 1




    $begingroup$
    Just leave it as CSV you don't need to do anything
    $endgroup$
    – arhwerhwe
    yesterday






  • 1




    $begingroup$
    Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ...
    $endgroup$
    – n1tk
    yesterday
















  • $begingroup$
    When I want to got 5 mts in distance, I would rather walk than to take a car.
    $endgroup$
    – Kiritee Gak
    yesterday










  • $begingroup$
    I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast.
    $endgroup$
    – honar.cs
    yesterday






  • 1




    $begingroup$
    Just leave it as CSV you don't need to do anything
    $endgroup$
    – arhwerhwe
    yesterday






  • 1




    $begingroup$
    Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ...
    $endgroup$
    – n1tk
    yesterday















$begingroup$
When I want to got 5 mts in distance, I would rather walk than to take a car.
$endgroup$
– Kiritee Gak
yesterday




$begingroup$
When I want to got 5 mts in distance, I would rather walk than to take a car.
$endgroup$
– Kiritee Gak
yesterday












$begingroup$
I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast.
$endgroup$
– honar.cs
yesterday




$begingroup$
I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast.
$endgroup$
– honar.cs
yesterday




1




1




$begingroup$
Just leave it as CSV you don't need to do anything
$endgroup$
– arhwerhwe
yesterday




$begingroup$
Just leave it as CSV you don't need to do anything
$endgroup$
– arhwerhwe
yesterday




1




1




$begingroup$
Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ...
$endgroup$
– n1tk
yesterday




$begingroup$
Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ...
$endgroup$
– n1tk
yesterday










3 Answers
3






active

oldest

votes


















4












$begingroup$

With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.



Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.






share|improve this answer









$endgroup$








  • 1




    $begingroup$
    +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
    $endgroup$
    – Bilkokuya
    yesterday











  • $begingroup$
    That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
    $endgroup$
    – B Seven
    yesterday


















3












$begingroup$

You can find a nice benchmark for every approach in here.



enter image description here






share|improve this answer









$endgroup$




















    1












    $begingroup$

    Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.






    share|improve this answer









    $endgroup$












      Your Answer





      StackExchange.ifUsing("editor", function ()
      return StackExchange.using("mathjaxEditing", function ()
      StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
      StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
      );
      );
      , "mathjax-editing");

      StackExchange.ready(function()
      var channelOptions =
      tags: "".split(" "),
      id: "557"
      ;
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function()
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled)
      StackExchange.using("snippets", function()
      createEditor();
      );

      else
      createEditor();

      );

      function createEditor()
      StackExchange.prepareEditor(
      heartbeatType: 'answer',
      autoActivateHeartbeat: false,
      convertImagesToLinks: false,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: null,
      bindNavPrevention: true,
      postfix: "",
      imageUploader:
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      ,
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      );



      );













      draft saved

      draft discarded


















      StackExchange.ready(
      function ()
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48008%2fwhat-is-a-good-way-to-store-processed-csv-data-to-train-model-in-python%23new-answer', 'question_page');

      );

      Post as a guest















      Required, but never shown

























      3 Answers
      3






      active

      oldest

      votes








      3 Answers
      3






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes









      4












      $begingroup$

      With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.



      Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.






      share|improve this answer









      $endgroup$








      • 1




        $begingroup$
        +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
        $endgroup$
        – Bilkokuya
        yesterday











      • $begingroup$
        That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
        $endgroup$
        – B Seven
        yesterday















      4












      $begingroup$

      With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.



      Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.






      share|improve this answer









      $endgroup$








      • 1




        $begingroup$
        +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
        $endgroup$
        – Bilkokuya
        yesterday











      • $begingroup$
        That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
        $endgroup$
        – B Seven
        yesterday













      4












      4








      4





      $begingroup$

      With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.



      Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.






      share|improve this answer









      $endgroup$



      With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.



      Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.







      share|improve this answer












      share|improve this answer



      share|improve this answer










      answered yesterday









      Shamit VermaShamit Verma

      1,024211




      1,024211







      • 1




        $begingroup$
        +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
        $endgroup$
        – Bilkokuya
        yesterday











      • $begingroup$
        That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
        $endgroup$
        – B Seven
        yesterday












      • 1




        $begingroup$
        +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
        $endgroup$
        – Bilkokuya
        yesterday











      • $begingroup$
        That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
        $endgroup$
        – B Seven
        yesterday







      1




      1




      $begingroup$
      +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
      $endgroup$
      – Bilkokuya
      yesterday





      $begingroup$
      +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
      $endgroup$
      – Bilkokuya
      yesterday













      $begingroup$
      That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
      $endgroup$
      – B Seven
      yesterday




      $begingroup$
      That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
      $endgroup$
      – B Seven
      yesterday











      3












      $begingroup$

      You can find a nice benchmark for every approach in here.



      enter image description here






      share|improve this answer









      $endgroup$

















        3












        $begingroup$

        You can find a nice benchmark for every approach in here.



        enter image description here






        share|improve this answer









        $endgroup$















          3












          3








          3





          $begingroup$

          You can find a nice benchmark for every approach in here.



          enter image description here






          share|improve this answer









          $endgroup$



          You can find a nice benchmark for every approach in here.



          enter image description here







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered yesterday









          Francesco PegoraroFrancesco Pegoraro

          60918




          60918





















              1












              $begingroup$

              Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.






              share|improve this answer









              $endgroup$

















                1












                $begingroup$

                Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.






                share|improve this answer









                $endgroup$















                  1












                  1








                  1





                  $begingroup$

                  Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.






                  share|improve this answer









                  $endgroup$



                  Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.







                  share|improve this answer












                  share|improve this answer



                  share|improve this answer










                  answered yesterday









                  MediaMedia

                  7,42262163




                  7,42262163



























                      draft saved

                      draft discarded
















































                      Thanks for contributing an answer to Data Science Stack Exchange!


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid …


                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.

                      Use MathJax to format equations. MathJax reference.


                      To learn more, see our tips on writing great answers.




                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function ()
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48008%2fwhat-is-a-good-way-to-store-processed-csv-data-to-train-model-in-python%23new-answer', 'question_page');

                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      -

                      Popular posts from this blog

                      getting Checkpoint VPN SSL Network Extender working in the command lineHow to connect to CheckPoint VPN on Ubuntu 18.04LTS?Will the Linux ( red-hat ) Open VPNC Client connect to checkpoint or nortel VPN gateways?VPN client for linux machine + support checkpoint gatewayVPN SSL Network Extender in FirefoxLinux Checkpoint SNX tool configuration issuesCheck Point - Connect under Linux - snx + OTPSNX VPN Ububuntu 18.XXUsing Checkpoint VPN SSL Network Extender CLI with certificateVPN with network manager (nm-applet) is not workingWill the Linux ( red-hat ) Open VPNC Client connect to checkpoint or nortel VPN gateways?VPN client for linux machine + support checkpoint gatewayImport VPN config files to NetworkManager from command lineTrouble connecting to VPN using network-manager, while command line worksStart a VPN connection with PPTP protocol on command linestarting a docker service daemon breaks the vpn networkCan't connect to vpn with Network-managerVPN SSL Network Extender in FirefoxUsing Checkpoint VPN SSL Network Extender CLI with certificate

                      Image
                      If human space travel is limited by the G force vulnerability, is there a way to counter G forces? Did Shadowfax go to Valinor? Do I have a twin with permutated remainders? Blender 2.8 I can't see vertices, edges or faces in edit mode 1960's book about a plague that kills all white people How can I make my BBEG immortal short of making them a Lich or Vampire? Alternative to sending password over mail? Is it inappropriate for a student to attend their mentor's dissertation defense? Why is Collection not simply treated as Collection<?> Took a trip to a parallel universe, need help deciphering Assassin's bullet with mercury Why is it a bad idea to hire a hitman to eliminate most corrupt politicians? I Accidentally Deleted a Stock Terminal Theme Is there a hemisphere-neutral way of specifying a season? What about the virus in 12 Monkeys? Why can't we play rap on piano? Arrow those variables! Today is the Center Modeling an IP Ad...
                      Read more

                      Cannot Extend partition with GParted The 2019 Stack Overflow Developer Survey Results Are In Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern) 2019 Community Moderator Election ResultsCan't increase partition size with GParted?GParted doesn't recognize the unallocated space after my current partitionWhat is the best way to add unallocated space located before to Ubuntu 12.04 partition with GParted live?I can't figure out how to extend my Arch home partition into free spaceGparted Linux Mint 18.1 issueTrying to extend but swap partition is showing as Unknown in Gparted, shows proper from fdiskRearrange partitions in gparted to extend a partitionUnable to extend partition even though unallocated space is next to it using GPartedAllocate free space to root partitiongparted: how to merge unallocated space with a partition

                      Image
                      Make it rain characters Keeping a retro style to sci-fi spaceships? How to split my screen on my Macbook Air? How can I define good in a religion that claims no moral authority? does high air pressure throw off wheel balance? Scientific Reports - Significant Figures Relations between two reciprocal partial derivatives? How to delete random line from file using Unix command? How did passengers keep warm on sail ships? How are presidential pardons supposed to be used? Are my PIs rude or am I just being too sensitive? He got a vote 80% that of Emmanuel Macron’s Change bounding box of math glyphs in LuaTeX Match Roman Numerals Was credit for the black hole image misattributed? How to pronounce 1ターン? how can a perfect fourth interval be considered either consonant or dissonant? Working through the single responsibility principle (SRP) in Python when calls are expensive High Q peak in frequency response means what in time domain? ...
                      Read more

                      Marilyn Monroe Ny fiainany manokana | Jereo koa | Meny fitetezanafanitarana azy.

                      Image
                      OlonaBiôgrafiaVangovangoMpilalaoMpilalao amin'ny horonantsaryMpamoaka horonantsaryTeraka tamin'ny taona 1926Maty tamin'ny taona 1962 EtazoniaLos Angeles fanitarana azy . Marilyn Monroe Avy amin'i Wikipedia Sauter à la navigation Sauter à la recherche Marilyn Monroe Ankapobeny Anarana Marilyn Monroe Teraka 1 Jiona 1926 Maty 5 Aogositra 1962 Fiaviana sy ny andraikitra Firenena Etazonia Asa : mpilalao mpilalao amin'ny horonantsary mpamoaka horonantsary Fiainana manokana Marilyn Monroe dia mpilalao, mpilalao amin'ny horonantsary, mpamoaka horonantsary mizaka ny zom-pirenen'i Etazonia teraka ny 1 Jiona 1926 tao Los Angeles ary maty ny 5 Aogositra 1962 Ny fiainany manokana | Ny vadiny dia James Dougherty, Joe DiMaggio, Arthur Miller. Jereo koa | Biôgrafia Rohy ivelany | Ao amin'i Freebase: [1] Mbola ambangovangony ity lahatsoratra ity ary tokony hofenoina . Azonao atao ny mandray anjara eto amin'ny Wikipedia amin...
                      Read more