Perl script to split a file and process then concatenate based on size and a string <p>Could somebody help me to get some logic to following in perl I am using windows 7.</p> <p>C:\script>perl split_concatenate.pl large_file a or b (Input would be large file and value a or b to process it later).</p> <ol> <li>Check the file if it is greater than 40KB (some size), and choice is "a" , if not run a command </li> </ol> <p>command -i large_file.txt -o large_file_new -a</p> <p>else if the choice is b </p> <p>command -i large_file.txt -o large_file_new -b</p> <p>else</p> <p>if it is greater say 40KB+, split the file for each 40KB arround (will be part1,) and append a first "particular string" which will be in the file_part2 to the part1 save it for processing, if there are multiple "particular string" then create subsequent files which should end with next "particular string" in the following part. ("Particular String" starts with some String but ends in different value). So script should search if there are more "Particular string", in the part2 or so and append first available one, if there is only one available no need to anything just split. As file always should end with a particular string. </p> <p>Then process same command </p> <p>command -i filepart1.txt -o filepart1.dat -a command -i filepart2.txt - o filepart2.dat (if needed) -a </p> <p>or </p> <p>command -i filepart1.txt -o filepart1.dat -b command -i filepart2.txt - o filepart2.dat (if needed) -b</p> <p>After this needs to be concatenated.</p> <p>Concatenate filepart1.dat + filepart2.dat + filepartN =large_file.dat</p> <p>I started to find the size first using below code,</p> <pre><code>#!/usr/bin/perl use strict; use warnings; use File::stat; my $filesize = stat("Full_File.txt")-&gt;size; print "Size: $filesize\n"; exit 0; </code></pre> <p>It will be great if some one help so I can learn. If this is not possible, then @ each 500th line the file reaches to 40KB, so I think this would be easier, every 500th line append the Next available "Particular String", and split and process above command, if file is less than 1000 lines then only 2 split and no need to append in the part2 as already it has one in its eof. May be easier? </p> <p>Better explanation:</p> <p>large_file.txt </p> <pre><code>xxx xxxx xxxx xxxxx var:value_var_v(1234) xxxxxx xxxxx xxxxx var:value_var_v(4567) xxxxxxx xxxxxx var:value_var_v(abcd) xxxxxxx // first split happens here as here assume it is 40kb xxxxxx xxxxxxx var:value_var_v(efgh) </code></pre> <p>If this is too big then split at line 5, say <code>large_files_part1</code>. Its end should contain <code>var:value_var_v(1234)</code>. After the 5th line it should split again at line 9 and will become <code>large_files_part2</code> and have <code>var:value_var_v(4567)</code> at the end. part3 wil go till line 12 and include <code>var:value_var_v(abcd)</code> at the end, and so on. If there is only one <code>var:value_var_v?</code> after the first split then only two parts is fine as long as the lines in both parts is arround 500. If there are say 1300 line in the main file then three splits are needed. The end of each file should have the next available "string", so the 1001st line will be the first <code>var:value_var_v(1234)</code> available after line 1000. String always starts with var:value_var_v, end with any thing. Hope this is clear.</p> <p>Output Expected: First case: So, out put will be, _part1.txt will be arround 40,000 if it had only one occurrence of string</p> <pre><code>xxx xxxx xxxx xxxxx var:value_var_v(1234) xxxxxx xxxxx xxxxx var:value_var_v(4567) xxxxxxx xxxxxx var:value_var_v(abcd) xxxxxxx // split happened here var:value_var_v(efgh) _part2.txt xxxxxxx xxxxx var:value_var_v(efgh) </code></pre> <p>After I do some process on these files (part1 and par2) I again concatenate</p> <p>_part1+_part2=large+file</p> <p>Final large_file after concatenation:</p> <pre><code> xxx xxxx xxxx xxxxx var:value_var_v(1234) xxxxxx xxxxx xxxxx var:value_var_v(4567) xxxxxxx xxxxxx var:value_var_v(abcd) xxxxxxx // split happened here var:value_var_v(efgh) **** xxxxxxx xxxxx var:value_var_v(efgh) </code></pre> <p>2nd splitting and concatenate case:</p> <p>If that file is too big say 80KB and has many strings "var:value_var() after a first split @40KB,do subsequent splits where it sees a next string which will be again "var:value_var_v()" and do a split, based on the string else based on the size. Everytime the file pat shoudl contain next available var:value_var_v().</p> <p>Orginal file:</p> <pre><code> xxx xxxx xxxx xxxxx var:value_var_v(1234) xxxxxx xxxxx xxxxx var:value_var_v(4567) xxxxxxx - - // assume now split happens here as here assume it is 40kb there are two more strings starting with var:value_var_v, split after var:value_var_v(abcd) and print this string in previous parts eof. Then final part will be ending with var:value_var_v(efgh). keep as it is. xxxxxx xxxxxx xxxxx var:value_var_v(abcd) xxxxxxx xxxxxx xxxxxxx var:value_var_v(efgh) part1.txt xxx xxxx xxxx xxxxx var:value_var_v(1234) xxxxxx xxxxx xxxxx var:value_var_v(4567) xxxxxxx - - // split happens here as here assume it is 40kb var:value_var_v(abcd) - //prints next available string which is var:value_var_v(abcd) _part2.txt xxxxxx xxxxxx xxxxx var:value_var_v(abcd) // Here part1 and part2 ends with same string. _part3.txt xxxxxxx xxxxxx xxxxxxx var:value_var_v(efgh) - This is last part and size should be below 40KB </code></pre> <p>process all of these part1,part2,part3 then concatenate to a big file.</p> <p>Final file after concatenating</p> <p>Full fille would look like.</p> <pre><code>xxx xxxx xxxx xxxxx var:value_var_v(1234) xxxxxx xxxxx xxxxx var:value_var_v(4567) xxxxxxx // split happened here in the first split assumed 40KB var:value_var_v(abcd) ****** xxxxxx xxxxxx xxxxx var:value_var_v(abcd) ****** xxxxxxx xxxxxx xxxxxxx var:value_var_v(efgh) ****** </code></pre> <p>PS: Once I get processed end of in each part I get **** a unique value and retain it as it during concatenation.</p>